Some states (Vermont), at least until 1918, and maybe after, held laws forbidding ex-Confederates to vote.
I'd say rebs up in Vermont were as rare as Baptist being sprinkled.