Jamie, I think the fear of "drowning in a sea of black faces" was in the back of the minds of many Southerners. Hints, at least, can be found in some of their wartime letters. I recall off-hand an example in a letter from an Arkansas captain to his wife, written after the fall of Little Rock, explaining his determination to fight on. He asked his wife to imagine their daughters sitting in school surrounded by blacks. Overtly racist by today's standards, but in the context of the times such familiarity between the races was considered by many to be improper, at best. He was concerned enough about it to cite it as one reason why he elected to stay with the colors, when many of his comrades were deserting and going back home after the terrible reverses of the summer of 1863.

Lee, of course, based his decision on far more fundamental grounds, involving loyalty and duty. I've read his correspondence on the subject many times over the years. It still brings tears to my eyes to read over the shoulder of a noble man being forced by circumstances to agonize over the most momentous decision of his life.