"If I were in enemy territory I wouldn't be walking around without my weapon in my own hands."
The event took place in Maryland, correct? Many Marylanders were sympathetic to the Confederacy, and didn't Lee hope to gain support from the locals? Overall, did most Confederates view Maryland as "enemy territory"?
(Also, after many miles of marching, I am sure a number of soldiers would have been happy to let someone hold their 10 pound musket for a while.)