In my personnal family history, I have such a man and story. He was concripted into the Confederacy, wounded at Prairie Grove Ar., and deserted to recover from his wounds, so the story says. But he was hiding from Bushwhackers so the story doesn't ring totally true. Then he joined the Union.
Now while he was a deserter, no one bothered his family, but after enlisting in a Union Regiment, they burned his farm to the ground, leaving my GGG an GG Grandmothers homeless.
I do not believe there was any threat to his family to join, he just didn't believe in secession.
Some Galvanized to get away from Prisons so terrible they knew they would die if they stayed there, so self-preservation play a role in it. Others just wanted to keep the Union and was forced to fight for the Confederacy. General Blunt reported hundreds of Confederate Deserters fought in Confederate gray for the Union at Prairie Grove Ar.
There was one group who I have never seen discussed in detail on this board and that was the peace activist, Peace Society.