1820-1840 was a period were Blacks were more and more excluded from rights and duties that, in the past, had been common. Several slave states allowed free Blacks to vote up until this period. I can only believe it was the threat of insurrection that worried these state governments.

David