Its too bad that the Vermont folks didn't explain that most of the Vermont Civil War records were burned in a fire in 1945, and all that is left are fragments. That's why they couldn't answer your questions or provide documents.
To be best of by knowledge, only Vermont officers were held at Libby prison; see http://www.vermontcivilwar.org/prisons/libby.php .
He was probably held at Belle Isle prison, and I haven't seen a list of prisoners from there. The fact that he was released at Wilmington, NC, indicates he was probably transferred from Belle Isle to Salisbury, Raleigh or Goldsboro, NC at some point. The April 22 1865 date is probably the date he was received back in Union lines.
Admitted at US Gent(?) Hospt. Div No. 1, Annap., Md(?) is actually US Gen'l (General) hospital, Annapolis, Maryland.
The affidavit from his hometown doctor indicates he was sent back to Vermont. A lot of veterans went home, briefly, then moved out west; there wasn't much of a job market in Vermont at that time (and lots of veterans got wanderlust having seen lots more of the country during the war than they had seen before, if they had even left Vermont.
So, I'd suggest you try looking for prisoner records at Florence, Salisbury, Raleigh or Goldsboro, NC.
If you do a google search for "March 1, 1865" Wilmington, you'll find dozens of references to men released on that date, having been at a variety of different prisons.
Hope this helps some.
webmaster, Vermont in the Civil War