Well, Frank, as Pam said ... look at the Agricultural census records. GG granddad owned land, horses, cows, a home, and a distillery. He lost his land, had his home and distillery burned, and his horses taken for Union cavalry mounts. That's what he did not have in 1870. And this was a man who did not even serve in the war. (He lived in divided Kentucky and I have Union vets among my ancestors.)
Slavery was wrong, and I certainly won't defend it. But it was legal at the time, and I was surprised to find a gggg grandfather who was a slave owner in Pennsylvania in the 1820s and '30s. As to free labor, it is strange (perhaps to some folks) that my family still knows their descendants.