When I was at Shiloh the pictures on the wall had Confederate soldiers with their "slaves" and Union soldiers with their "servants". I thought that was ironic.
The border states were not the only slave states. I will have to look it up, but I know there were slaves in some Northern states during the war. There were many plantation owners who lived in the Northern states and had their overseers run their Southern plantations.