Since you mentioned larger towns and cities, I must assume you're focusing on people other than families living together in single dwellings. Examples would be schools, hotels and construction crews. If anything, the skewing you described could make the number of slaveholding families appear fewer than they actually were. For example, hotel managers and teachers would not be more likely to own slaves than other adults. Also, visitors and students who came from slaveholding families didn't always take slaves along with them. More often than not, those people would be mistakenly counted as non-slaveholders.
Back before the advent of the internet, I spent three years working with the 1860 census of Barbour County AL. Barbour was favorable for study because it is typical of most Alabama counties, larger towns incuded Eufaula and Clayton. During this time I used the census to match members of households to companies recruited in and around Barbour County.
In many cases older sons who had married still lived close by their parents. Living separately, they would be reported as individual households. If the parents owned slaves but the sons did not, only one household would have been noted as slaveholding. Can we be certain that slaves owned by the parents didn't work on land owned by adult sons? I think not, but that's the normal rule used.