I suppose people have different definitions of loyalty, but I am certain there were Unionists who reluctantly supported the Confederacy. I read a biography of Mijamin Priest, a Unionist in East Texas. Born in PA, he moved to Alabama in early adulthood and he was at various times a lawyer, preacher and teacher. At the outbreak of the war Priest was Chief Justice of Rusk County. All four of his sons enlisted in the Confederate Army and two died at Camp Douglas. Priest kept his job as county judge until 1866, when he lost an election to a Confederate veteran. Oh, and he was also a slaveholder.