I think it is too tempting to read much into this statistic. I know there were slave holding Unionists in the South. They may have feared that war would bring them economic ruin and was more a threat than abolition. After the War began, they failed to change sides.

Also, Bryan Howerton educated me about some of the harsh consequences of Consription in Arkansas and how it antagonized families and communities until they became pro-Union. My wife's gg-grandfather was in the 4th US Cavalry (age 43), as were two brothers, one of whom was galvanized. One other brother deserted the CSA and another died 2 months after enlistment. They were immigrants from Alabama.