Bob --
Recognizing the points you made during the war, Northern war planners took steps to physically remove slaves that might contribute to the Confederate war effort. I don't believe anyone seriously brings your points into question.