I would think that if they were not buried in Port Royal that they would be buried in Fredericksburg and later moved to Spotsylvania Confederate Cemetery.
Do you have the name of someone you are looking for?