Ann,
You might try the prisons board. There are some very informed posters on that site with access to all the records.
For the first two years of the war prisoners were exchanged under the Dix Hill agreement. I was only after General Grant figured out that the south could not replace its captured personnel and the north could that the exchanges stopped. When the exchanges were in place most were only held for a couple of months then returned to duty.