Your correct that many loyal to the Union were far from being abolitionists. However your comments about Grant and Lee are incorrect. According to "Team of Rivals" Grant was given one slave by his father-in-law. He and Grant worked to build a rough cabin Grant named Hardscrabble. This house is now at Grant's Farm in St. Louis. Soon Grant went bust and was trying & failing at several jobs in St. Louis. It's said that he sold his watch to buy Chistmas gifts. It was at this exact time Grant Freed his one slave because he was no longer working a farm. Grant could have easily sold his property "Down South" for a substantial amount of money, but he did not.

Lee owned slaves, although most were in his wife's name. According to Charles Flood's "Lee: The Last Years" Lee often returned to his plantation early in the war to oversee detailed instructions he had provided. Lee thought his slaves were taking advantage of his absence and ordered punishment for those who were slacking. Lee did however, after he surrendered and was living at his home under what he thought would be his arrest for treason at any time. While at church one Sunday the preacher called for people to kneel at the alter to partake of communion. To the surprise of the congregation, a black man sitting with the other negros kneeled alone at the alter. With a silent shock in the church, it was Lee who rose, walked to the alter, and kneeled next to the former slave.

Our country owes more to these two men than most realize today.