I don't doubt there was some concern because of the memory of past uprisings but were there not also blacks employed by the Confederate states who did factory work, lumber mills, dock work and such? I know here there were slaves and also free blacks employed here to 'shore up' Ft Barrancas. They also did other things at the Navy Yard. I'd think even if they were not brought into the military as soldiers (in the early years), they did replace former white workers for some things. At some point, even those who were against it in the beginning had to see the value of the idea when it started looking rather bleak.

Pam