For a moment, consider that William T. Sherman wanted nothing to do with U.S.C.T. units and did not allow them in his army in Georgia and the Carolinas.
Back to the slave-holding classes, the typical slaveholder in Alabama in 1860 owned seven slaves. Another perspective, half of all households that included slaves counted from one to seven slaves. That's not exactly Tara or Twelve Oaks. As William C. Oates stated, "the great majority of the men in the ranks, those who handled the muskets and did the killing, were not of that class."
I hesitate to quote this disabled Confederate veteran further because of the impact his words have on some people. Right now I may hold on to further quotes as a 'nuclear option.'