Clay, sometimes pows were allowed to be in their own territory on parole until they were exchanged. Both governments set up parolee camps to hold the men until the exchanges were made. I believe Enterprise, Alabama, was the parolee camp for Confederates parolees from Vicksburg. However, General Grant allowed a lot of pows from the Trans-Mississippi West to go home after being captured at Vicksburg because he felt that many of them would never report back to duty when exchanged. Your man might have been one of those.