That's well put. Slavery carries such a stigma that folks want to dissociate it from the period Southern culture and economic and morality decisions. If I say an area had a lot of slaveholders, then folks from that area or who had ancestors in that area unfortunately too often consider that an attack on them or their kin. It is not. It's not much different than if I said they raised a lot of tobacco or cotton.
My suggestion is to think of slavery in cold economic terms. If slavery was legal and in the period could be morally rationalized in some way, then it would be used in areas that made it a profitable endeavor. It would be less frequent in areas where slaves were more of a luxury than an economic multiplier. Let's face it, most of the time humans (including Americans) will eventually rationalize what is in their personal best interest and/or survival. So rather than try to classify them as good or evil based on slave ownership from a modern perspective, think of it more in terms of the 1860 economic decisions and background.