Non slave holding white people probably benefited in some respects, but not generally. If the plantation owner used slave labor to build a grist mill, for example, others would benefit indirectly for having the grist mill in the community.

There was a social fear, North and South, about assimilation of free black people into society. It was the genesis for the black codes in some Northern states and a major reason for the political fight to admit so-called "Free States" into the Union. In the South, slaves were not a threat to white people and not seen as a benefit to non slave owning white people. They were neighbors, usually on good terms. It was not the end of slavery that presented the social upheaval so much as the method and the punitive Reconstruction that unnecessarily made blacks and whites compete for survival.