You notice how many more Union deserters are listed than Confederate?

A number of these were bounty jumpers. There are records of some who would enlist 6-7 times for the bounty and desert each time. Some Southern sympathizers enlisted as a means to get to the South, then joined in the fray on the Southern side.