I had read several comments and quotes of Lincoln and came to that same conclusion some time ago. I think most people want to still hang on to the slavery issue, and disregard the fact it was simply a matter of race regardless if they were slave or free.
More to the point I think the issue of constitutional rights were at stake. If the government could stop a man from bringing his property to the new territories, what other laws could be passed that abused his rights?
GP