Many Confederate soldiers who happened to be there were home on furlough. Large numbers of Confederate prisoners had recently been released on exchange and furloughed from Richmond VA. Others had been detailed to the public works at Columbus GA.

I have not read the book but a substantial number of Confederate troops were involved in the defense of the city, which took place largely on the Alabama side of the river. There was no U.S. military prison at Macon GA. The city had always been in Confederate hands up until it surrendered to Gen. James H. Wilson on April 20th. General Wilson had no intention of staying there and quickly released his prisoners on parole.