One source often overlooked is the 1860 Federal Census. I realize that a lot of people moved around in the War years, bujt you might get a good handle on general populations of towns/localities by checking the census. And people are usually listed by household.
Many public libraries subscribe to Ancestry.com and Heritage Quest, both of which have available the 1860 census.