Here's the actual account:

At Williamsport the following amusing incident occurred. While the wagon-trains were massed there

Page 190
waiting for the river to fall, the enemy's cavalry approached and shelled the banks of the river. There is a deep hollow or depression there on the north side of the Potomac, and here the wagons were parked. A Confederate quartermaster officer approached the spot during the artillery fire and was amazed to observe that not a single teamster was to be seen. He could not account for it, until he happened to look toward the river, and there saw hundreds of black heads just showing above the water. The negro teamsters with one accord had plunged into the river to escape the shells, and were submerged to the neck!

pgs 189-190 of Randolph McKims A Soldier's Recollection
http://docsouth.unc.edu/fpn/mckim/mckim.html

He states this occurred where the "wagons were parked."

I would therefore believe these were not the armed Wagoners who had been formed in the ranks on the two flanks of Imboden's defense line.