It probably depends on how or if you choose to count the various militia and irregular formations. Overall, far more men joined Federal forces (whether they liked it or not!). I would say well over 100,000 for the North and around 50,000 for the South. What do the experts have say?