If I remember correctly, thousands of Union troops deserted when news of the EP got to camp. At least they were being paid and their families were not homeless and starving. Noone had destroyed their homes and stolen their food.
When you get word that your family is in dire straits and needs you at home, what will you do if you are a real man? A real man will take whatever risks necessary to reach his family. God, family, country, in that order. That is how it should be.