Along with the future of the territories, you had the abolitionist trying everything he could to get a slave insurrection, helping slaves escape and totaly disobeying Federal Slave Laws. It was very frustrating times for the Southerners.

More importantly you had bigotry, it wasn't because they didn't want slavery in those territories, it was they were afraid the slave owner might free those slaves and the Northerner would have to live around those black people. Like I said before, the Southern States proposed an emancipation program in 1827 that the Northern States would have no part of. So no matter how a person might wail about slavery, they still had that "not in my backyard attitude".