Checking the Richmond City map ca 1860s, I'd have to believe your ggrandfather was buried in the Hollywood Cemetery.
I find a number of men who died at Jackson at that time were buried at Hollywood. See map at:
Click on the lower left corner to get a blow up.
Jackson Hospital was to the SW of Hollywood whereas Oakwood Cemetery is way off to the NW of the city.