Probably Dibrell Warehouse Hospital
Opened in June 1862, irregular usage. Northwest corner 20th and Cary Streets.

Or it could be the Danville Railroad Prison
http://www.censusdiggins.com/prison_danville.html

Libby Prison Hospital
Located in a room in Libby Prison for use of inmates. John Wilkins in charge to mid-1864. Dr. George William Semple, afterwards. Enlarged and operated by Federal occupation forces as John Wythe Hospital.

http://www.mdgorman.com/Hospitals/hospital_index.htm

http://www.censusdiggins.com/danville_prisoners.html