Only a small percentage of Southerners owned slaves and in their agriculture based economy, poor and mariginal white people often competed against slave labor. The overabunance of slaves had depreciated the value of owning slaves and so did the cost of feeding them (keeping them healthy). Increasingly, only farmers with large cotton farms were reaping significant financial benefit from owning slaves. Slavery, in the United States, was on the way out. There never was a large demand for slave labor out West and the large influx of European immigrants and the push of white people people to move westward, destroyed any notion forever.
The complaint that Southerners had against Northern politicians and federal policies were legion. Property rights was one of many concerns along with unfair taxation and under funding of projects, such as railroads. In Texas, the citizens were constantly being attacked by hostile Indians on the frontier and Mexican bandits, with very little of the army support promised by the federal government at the time of annexation.
The Southern States simply did not have any power to abuse the rights of Northeners. The facts portray quite the opposite.