In 1602 the Spain had to decide what to do with St. Augustine should they move the colony, Abandon it, or keep it? The decision was to keep St. Augustine, as a haven for shipwreck victims and a military position against the increasing threat of pirates preying on the Spanish treasure fleets. With the arrival of Governor Mendez de Canzo started many improvements on the city, the establishment of a hospital, a plaza and market, and a new block of housing on the southern tip of town.
Even with all of these changes St. Augustine became more isolated, poor and neglected by Spain than it had been. Wars, threats to holdings in America, and the loss of some of their treasure fleets made St. Augustine a low priority of the crown. In 1633 St. Augustine had yet another governor Louis de Horrutiner he only lasted five years, on the year of his departure a hurricane hit the city.
In 1655 Governor Don Diego de Robelledo arrived. In 1658 the city was the object of pirate attack this time by the Pirate Robert Searles. In 1671 yet another new governor arrived in St. Augustine Sergeant Major Manuel de Cendoya. Also in that year the coquina quarries open on Anastasia Island and the first stone of the Castillo de San Marcos was laid.
The first lighthouse was built in 1673 on Anastasia Island. The next year a hurricane hit and leveled the city. Governor Juan Marquez Cabrera arrived in 1680. During his time in power he started the first black militia unit. They were put to good use against pirate attacks in 1683 and 1686 both attacks failed. At the close of the 1600's the Castillo de San Marcos was finished and a sea wall had been constructed to help prevent flooding.