"geography determined the side they fought for"

Actually after early 1862 it was the Conscription Acts (draft laws) in both the North and the South which determined the side most soldiers fought under. It was worst in the South.

Depending on whose army occupied what territory and how strong their presence in those areas was, those conditions usually was a predictor of which side an able bodied man fought with.

If a patrol rode by your farm and asked you why were you not in service, it was pretty hard to resist their invitation to join them. Many southerners were enlisted into the Federal army, when being confronted by a union patrol and being accused of being a Confederate deserter because he was of military age and able bodied, was offered only two choices Prison or enlist.

Which would you choose?

All in all the idea after 1862 that a person had a right to choose which side you wished to fight on, is a myth. If you were a northerner and fought in the southern army, you were as subject to being shot as a traitor if captured as you were if you were a southerner fighting in the Union army.