You are probably right.

But, from what I've read about the early reconstruction carpet bagger governments they didn't need much of a reason to make southerners pay for their "crimes". The fact that you had lived in the south at all during that time was reason enough.