Hi Linda -
don't be too awfully hard on yer ancestors in the Union Army. I found out that one of my gg-grandfathers from Southern Illinois served in the Union 48th KY and 26th KY Infantries (see URL below).
The saving grace is that his family and inlaws had moved to "Little Egypt" from Tennessee and Kentucky. Several of their kin did indeed fight for the CSA; wonder how often they were on opposite each other....hmmmmm.....