I have a letter written by a non-slaving holding Confederate soldier who wrote a letter to his wife in which he stated something along the line of (it's not in front of me so I can't quote it exactly) "we can't lose this war. Can you imagine if we fail our little Suzy having to share a school desk with one of them?"

He didn't own slaves or money or land. The one thing he did have was the status quo which was the fact that he was better off than the black man.