My GGrandfather (41st Alabama Infantry) was a prisoner at Point Lookout, MD. Luckily he was only there from Late March to mid June 1865. The prison guards at Point Lookout were from the US Colored Troops and spent much of their time taunting and trying other means to get a response from the prisoners. If they could get a response, they then shot the prisoner. They had the saying "The bottom rail is now the top rail" meaning they expected to extract revenge for perceived injustices. Many stories of the cruelty of the US Colored Troops guarding Southern soldiers.
There is even a Point Lookout POW Descendants Organization.
http://members.tripod.com/~PLPOW/plpow.htm