Dennis,
Quite a few were actually carried to Key West during the war. Pretty much every prisoner taken by the Army or Navy up the Gulf Coast as far around as St. Andrews Bay was taken to Key West and usually sent on from there to other Northern prison camps.
The prisoners taken during the Natural Bridge and Ricco's Bluff expeditions in 1865 were all taken to Key West.
Best,
Dale