There is a Robert L Williams in Dyer's Compendium listed in Co E 6th TN Cavalry USA, Co E, pvt, age 26 enlisted on 04 Feb 1864 (that would presumably be when he enlisted in the 13th/14th). Under remarks it says Returned POW, sick. This would fit your diary account that he was captured at Ft. Pillow and sent to Andersonville. He could have been released when exchanges began again, then placed with his former 13th/14th comrades who were in Co E 6th by that time.