"If the men were that worried that there were a credible threat from the slaves to their family and home why would they have ever left to go fight a war?"

There were still home guards and militias at home made up of men who were not capabile of hard military service for such disqualifications as age or disability. These men gave a reasonable sence of security and maintained law and order on the home front.

During reconstruction Blacks were still used as a weapon against southern white by the federals as recipiants of confiscated lands and as a voting block to elect and keep in office the Carpet bag governments and even being a part of those repressive governments passing repressive laws that in some cases were not repealed until the 1950's and some may not have been repealed to this day. Black soldiers often were used to maintain military control over the political districts. The point of a bayonet is not the way to win friends. This did nothing to cement favorable race relations in the post war South and only breeds resentments.

I am not sure that Lincoln, due to his illness (medical poisoning), would have survived another year anyway and doubt that he would have had the stamina to resist a radical Congress out for revenge against the South.