Folk seem to forget the "Black Republicans" were a big source of concern for the pro-slavery South. The South was convinced that the "Black Republicans" were the big threat, and when their candidate Lincoln won the presidency, it was time to go.
Kevin Dally