Ken,

Captain Augustus H. Drewry was the commander in 1862, and the fortifications were manned by the Southside Artillery, plus the crew of the C.S.S. Virginia (scuttled on On May 9,1862, when Norfolk fell to Union forces). Commander Ebeneezer Farrand then supervised the defenses of the fort. I have not found the record of the artillery unit that was in place during 1864.