While recently reading Kent Brown's, "Retreat From Gettysburg" I am surprised to note the estimated number of Black men that accompanied Lee's Army of Northern Virginia into Pennsylvania. 12,000 - 20,000. Half with the reserve train gathering supplies and half with the actual army.
Tho some few did take this opportunity to run away, the vast majority did not. Great numbers most certainly could have escaped and is probable were not under armed guard.