In 1843, Major Robert Gamble Jr. established a sugar plantation along the Manatee River, a region then remote from civilization. His mansion took six years to build. Gamble accumulated almost 3,500 acres, but natural disasters and a fickle sugar market drove him into debt by 1856. He sold the plantation in 1859. At the end of the Civil War (1865), the U.S. government ordered Confederate Cabinet members arrested. Judah P. Benjamin, Confederate Secretary of State, fearing trial for treason, escaped by traveling through Florida. He is thought to have briefly sheltered at the mansion.
Benjamin escaped and traveled to England, where he became a barrister. In 1925, the United Daughters of the Confederacy purchased the property and deeded it to the state of Florida. Gamble Mansion and Patten House have been restored to the appearance of their respective historic periods.