That my friend was the $64,000 question that the south was asking. So you emancipate 4,000,000 slaves what is to become of them? And the north had no answers.
By the way you have heard the southerners favorite Bible Scripture haven't you?
Isaiah 43, verse 6:
"I will say to the north, Give up; and to the south, Keep not back: bring my sons from far, and my daughters from the ends of the earth;"