Reviewing what was going on at the time, 5th Vt Inf was a part of 4th Corps and they were involved in the Peninsula Campaign in Virginia in 1862. The Primary battles were the siege of Yorktown, April 5th to May 3rd and the Battle of Williamsburg, on May 5th. POW's from these battles would have gone to Richmond, probably Libby Prison.

In June 1863, the 5th Vt Inf was involved in Franklin's Crossing, Rappahannock River where, on June 5th, the 26th New Jersey & 5th Vt, 15th & 50th New York Engineers, supported by the 6th Corps attempted to cross the Rappahannock River at Franklins Crossing.

Captured here, he would have gone back to Richmond and Libby Prison, but his compiled military records should give you all the information you're looking for.

JR