William Madison Scott was a kid that served with Rayburn and for what it's worth he wrote in his memoir:

"Some historians tell us that Doc. Rayburn died of tuberculosis, this is true, but it was while a prisoner in the Federal prison that this disease was contracted. When surrender came, Rayburn and His men were in a mountainous country, almost inaccessible and they would not come down. The Yankees sent word if they would come down and surrender, take oath of allegiance, they would be allowed to go home. Rayburn, taking them at their word, brought his men down and surrendered. All were allowed to go except Rayburn, he was thrown into prison and kept almost a year with no pretense of trial. When he became so emaciated from long confinement that he could not get about, they put him on the train and sent him to Duvall's Bluff. I happened to be at the station when the train came in and I took him in my arms and placed him in a carriage and he was taken to the home of his sweetheart where he shortly passed away in the flower of his manhood, just another victim of the atrocious prisons of the North."

I seem to recall the little book "Rebels Hell: Little Rock's Yankee Prison" (I believe that's the name of it) has something about Rayburn in the front of it...maybe it offers a clue.