With all proper deference to Capt. Jack Sparrow and the Pirates of the Caribbean franchise, there’s a far richer (and bloodier) vein of real pirate history and lore running through the islands of the Caribbean. And while some of it has literally sunk into the sea, there’s still plenty of opportunity for swashbuckling travelers to walk in the boots of some of the Caribbean’s most infamous buccaneers.

Kingston, Jamaica

In all the world there was no more notorious pirate town than Port Royal, which once stood on the thin stretch of land known as the Palisadoes at the outer edge of Kingston Harbour. At one time the largest and richest city in the Caribbean, Port Royal was a favorite hideout for pirates thanks to its close proximity to the trade routes where they found merchant ships and the Spanish Main to prey upon.

By the mid-1600s, the swelling population of pirates and privateers based here made Port Royal — dubbed the “Sodom of the New World” — the de-facto capital of seagoing pillage in the Caribbean. However, a devastating earthquake and tsunami in 1692 destroyed two thirds of the city, with much of it falling into the sea. After a series of subsequent fires and hurricanes, Port Royal was largely abandoned. Today, visitors to Kingston can tour Fort Charles, which dates from the 17th century and includes some relics retrieved from Port Royal, and snorkel over the ruins of the sunken city, which is under consideration as a UNESCO World Heritage Site.

Nassau, Bahamas

In its pirate heyday, Nassau was nearly as notorious as Port Royal. Once home to more than 1,000 pirates, including the infamous Edward Teach, a.k.a. Blackbeard. New Providence Island became known as the capital of the Pirate’s Republic in the 17th and early 18th century, until the British Royal Navy attacked the island and chased off or rounded up the resident pirates. The end of Nassau’s pirate reign was marked in 1718 by a mass execution of pirates in the town square.

The Pirates of Nassau does a fine job of detailing the island’s brigand past and features several immersive displays, including a recreated quayside with a pirate ship at anchor. The Graycliff hotel in downtown Nassau was built in the 1700s as a home for pirate John Howard Graysmith. The hotel’s world-renowned wine cellar, home to the third largest private collection in the world, was once a pirate dungeon.

Most of Nassau’s surviving historic sites, including its three forts and the famous Queen’s Staircase, were built after the reign of pirates ended. But the Deanery House on Cumberland Street is believed to have been built in 1710.

Tortuga Island, Haiti

The 69-square-mile island of Tortuga, which lies off the north coast of Haiti, traded hands between the French, Spanish, and English several times during the 17th century, but when it wasn’t being occupied or invaded by one navy or the other it became the haven for the Brethren of the Coast, an alliance of pirates and privateers led by legendary pirate Henry Morgan. Today, the island is still associated with dark commerce, as a departure point for smuggling trips taking Haitians to the shores of south Florida. Tortuga is seen as having huge potential for tourism — Royal Caribbean was even thinking of building a cruise port there about a decade ago — but the ongoing turmoil in Haiti has largely kept visitors away.

Dominica

The fierce Kalinago people inhabiting Dominica kept the island free of European conquest well after most other Caribbean islands had succumbed, making it an attractive place for pirates to hide out, repair their ships, and trade with the local inhabitants. Dominica remained largely overlooked by visitors until the Pirates of the Caribbean movies were filmed in several locations during the early 2000s, and visitors can tour sites like the Titou Gorge seen in the movies.

St. Thomas, USVI

Blackbeard made his smoky presence known far beyond Nassau: he also spent time with his band of brigands on St. Thomas, where Blackbeard’s Castle — part of what was for years a hotel — is reputed to have been used by the pirate as a watchtower. Erected by the Danes in 1679, the 40-foot-tall hilltop tower was originally called the Skytsborg, but given its age and location it’s certainly possible that Blackbeard climbed its 99 steps to scan the horizon for approaching ships. The list of pirates who operated in the Virgin Islands is long, including such legendary captains as William Kidd, Jean Hamlin, Stede Bonnet, Tempest Rogers, Bartholomew Sharp, and Black Sam Bellamy. Be sure to check out the Pirate Treasure Museum, which includes a collection of real sunken treasure recovered by marine archaeologists.

St. Croix, USVI

As part of the U.S. Virgin Islands, St. Croix is known for its Danish history. But French pirates were thought to be operating from the island as early as 1522, which not only predates the arrival of the Danes but would make St. Croix one of the first pirate colonies in the Caribbean. The site of the Buccaneer resort was once the home of privateer Jean Martel, and the remains of several pirate ships sunk by the British Navy in 1716 lie in the waters of Salt River Bay.

British Virgin Islands

You know that pirates plied the waters of the BVI for one obvious reason: Normal Island is actually named for a pirate, as is Bellamy Cay and Sir Francis Drake Channel. The island also was one of the few places in the Caribbean with a documented history of having been a repository for buried treasure (stashed there by the mutinous crew of a Spanish ship in 1750), and served as inspiration for Robert Louis Stevenson’s classic book, “Treasure Island.” BVI visitors can visit Norman Island, dine at a beach restaurant, and hike to the high point of the island, but the hottest action is just offshore, where the party boat Willy T sits at anchor in Norman Bight. (An earlier iteration of the boat, sunk during a hurricane, is a popular dive spot inhabited by a skeleton crew of pirates.)

Cayman Islands

Who says piracy can’t be fun? Every November, the Cayman Islands celebrates Pirates Week with mock pirate invasions, costume parties, food festivals, and more. The whole thing has become so popular that it has been expanded into a seven-week-long Pirate Fest. In a way, the hospitality mirrors some of the things that attracted the pirates to the remote Caymans in the first place: fresh water, turtle meat, and wood. Blackbeard, who apparently was everywhere (and nowhere) in the Caribbean, occasionally ported here, as did fellow pirate Edward Low, a particularly nasty piece of work known for torturing prisoners to death.