I am not real familiar with the Kinston area. I live 3 hours away in Elizabeth City. I have had good success at getting photos of grave markers through Find-a-Grave. If you know the name of the church, I have a friend that is a member of Find-a-Grave that will put out a call for a photograph. I got photos of 5 grave markers of men that died while imprisoned on Governor's Island within a couple days after he submitted a request.
I am currently featuring photos of Governor's Island on my web site. Fort Warren will follow. There is a nice UDC memorial marker at Fort Warren dedicated to the Hatteras prisoners and others that died there. I would love to have more photos if I could post them on my web site.
Captain Thomas Sparrow's diary of his time at Hatteras and while in captivity is now online at: http://www.lib.unc.edu/mss/inv/s/Sparrow,Thomas.html
Sparrow commanded the Lenoir Braves. The men fortunate enough not to get captured traveled up the Neuse River to New Bern, where they served with the 40th NC at Fort Lane.