This is starting to make sense. I never knew he was drafted into the Army, but this would explain why he didn't enter service until '63, when he was of age before the War broke out. I think Norfolk came into Union possession sometime in 1862, when the Confederates pretty much evacuated to the Peninsula. I believe the Confederates burned the shipyard on their way out of town.
Thanks again for your input Fredrick!