You could put it that way David, or the numbers speak for Pro-secession and anti-secession. I believe alot of it was the anti-secession group, but their arguments went along with anti-slavery too. I read a flyer recruiting from Winston County Alabama, for the 1st Alabam Cavalry, that spoke out against fighting so the slave owners could own slaves along with proclaiming loyalty to the Union.

With this in mind I believe the Southern Unionist, was too poor to own slaves and felt the secession was only to further slavery.