"but these men knew that if the Union won, they would be free."
It is to my way of thinking that regardless of who won they would be free. Don't think the US or the CSA would want an internal civil war (if) after fours years of fighting the war ended in a stalemate.