The officers who left the Union Army were "on duty" when they resigned and they did join another army to "kill some of their own" as soon as they came to disagreed with their superiors. The vast numbers of enlisted men who deserted (both armies) did not do so to join the opposite army or kill their former comrades. They too came to disagree with their superiors but had no recourse like resigning. You are certainly welcome to consider the double standard trivial.