New Bern, North Carolina, was captured by Federal forces fairly early in the war (March 14, 1862) and remained in their hands to the end of the war. It was used as a naval station and supply depot by the Union, but as far as I know there was no military prison there. That's not to say, however, that relatively small numbers of prisoners of war weren't held there. It may also have served as an alternative exchange point in the very late stages of the war, after Richmond and Petersburg were captured.