I have been reading Rev. Agnew's diary on line at the link that Mitchell provided. His Dec. 17, 1863, entry mentions a "Parson Vandiver." If this is the same man, perhaps he was released and, being disabled, was allowed to return home unmolested.