What is your ancestor's name and unit of service? He was likely released from Point Lookout in February 1865 as a paroled prisoner of war. It would have been unusual for him to be released upon taking the Oath of Allegiance and allowed to "go home".

Hugh Simmons
Fort Delaware Society
E-Mail: society@fortdelaware.org