They had good reason to feel that way. They had witnessed what happened with sudden emancipation in the Dominican Republican and the complete collapse of that country, EVEN TO THIS DAY. They had already witnessed The Nat Turner rebellion, and John Browns Raid. And they were well aware of the Aggitators in their midst, even including at least 7 northern Governors that openly refused to obey the laws of the land in open defience of the Federal Government.
But, "Fearing the Blacks" wasn't the reason a man like Lee resigned the Army. "Freeing the Slaves" wasn't the reason that started the War that lead to Lee resignation in 1861. That "Battle Cry" was two years later.
The reason I believe that Lee resigned was that The Constitution of the United States had become a worthless document, it was not being observed as a guiding document, just like we are doing today. Therefore there in effect was "NO UNION" because of that, and that it was every State for themselves because the Federal Government was violating the restriction place on it by the Constitution nor would it enforce those provisions which protected the soverign states.
Remember Lee's oath as and Officer in the United States Army was to the Constitution, Not the United States.
I don't belive that this article makes any mentioning of any of this. For a good understanding of this read the 1st Chapter of Captain Augustus Dickels, 3rd South Carolina Inf. book "History of Kershaws Brigade."