In my 2gtgrandfather's case (David Pierpont MORGAN), and I have NO documentation to back it up, I rather think it may have been, at least partly, due to the fact that his father Richard MORGAN, had been a Rev. Soldier from Duplin,NC. His wife, Frances Watson, was the grandaughter of Thomas GIBSON, who fought with Francis Marion. With such close ties to the recently fought Revolutionary War, I think there were those who just couldn't bring themselves to fight to destroy what their ancestors had so recently brought together.
There were also a lot of our men who valiantly joined the CSA, wound up God knows where, and were refused parole when they rec'd word of a wife or child's severe illness, coupled with hearing others talk about the confiscation of foodstuffs. This is why many of them deserted...not from cowardice, but from the conviction that they were needed at home taking care of their families.
Other thoughts?? It's always fun to see what others think, or know. Mr. Pitts knowledge is awesome!