You're right about the number of factors that influenced Unionism in Panhandle Florida and the border counties of south Alabama. The breakdown of tansportation and markets during the war caused many families to go hungry for extended periods. Security from roving bands of outlaws and ability to feed their families influenced many of these men. First class arms, equipment and uniforms represented other incentives for enlistment that Confederate recruiters could not match. Regular pay in U.S. currency also weighed heavily in favor of Federal service.

As you mentioned, it helped that troops recruited from Florida and south Alabama for U.S. service were retained near their homes.

However, it's important to note that many well-to-do slaveholders in Florida, Alabama, Mississippi and Louisiana did everything possible to oppose secession and keep their states in the Union. Alfred Holley of Covington County AL is just one example.

Discontent over the Emancipation Proclamation provoked many Midwestern farm boys to desert U.S. service. Of course none of them owned slaves. Farm boys from the Panhandle might have reacted to emancipation similarly. Just because a man lived in a household without slaves, we can't assume that he was hostile or even indifferent towards slavery.

To answer the original question, one common thread for Unionism in the lower and mountain South appears to be isolation. It's somewhat ironic, because people usually associate Unionism with industry, commerce, modernization, diversity and freedom. The same people usually link the Southern cause with agriculture, traditional folkways, backwardness, homogeneity and slavery.

In actuality support for secession and the earliest military companies raised for the Confederacy came from market centers like Mobile and Pensacola. These cities and smaller towns along rivers and rail lines exhibited rapid growth, prosperity based on commerce and immediate connection with the outside world, a high degree of literacy and culture. They also show evidence of widespread social and political involvement as well as a diverse population. Print media in the form of daily and weekly newspapers provides another important indicator, in that they helped to disperse ideas and opinions.

In contrast, Panhandle counties that provided recruits for U.S. military service were quite isolated and backwards. The 1860 census of those counties show that they had few merchants, attorneys, clerks or teachers, small market centers with limited connection to the outside world, hardly any sign of cultural, educational or literary life, indifference towards social and political activities outside the immediate community, resistance to change and no newspapers. These characteristics can be applied to Unionists found in isolated communities across the South.