In Dickert's book, "The History of Kershaw's Brigade" the author clearly states that slaves accompanied Lee's Army INTO Pennsylvania. These were the Officers body servant. Dickert gives a rather interesting account of the servants (Including his own's) activities ranging far and wide to gather food for their master meals and that you usually did not see them during the day as the columns moved, but would catch up to the column at night.
I know that the popular idea is that nothing of this sort happened. Especially a slave marching north into Pennsylvania and then returning back to Virginia as Lee's Army retreated. But that is exactly what Dickert's says happened with only a couple of exception of where the slave became losted and seperated in Pennsylvania.
So my thought is that IF this was happening, (and we don't know to what extent that this was taking place) it would appear to the outside observer that these slave were being gathered up as Lee marched throught Pennsylvania, and were being return to their former state of servatude. When in fact they had accompanied Lee's army to Pennsylvania in the first place.
So given this account in Dickert's history of Kershaw's Brigade, we can not mearly follow the popular assumtion that the blacks observed accompaning Lee's army were ex-slaves being gather up by the Confederates as they moved through Pennsylvania.