On the 1850 census a Kelly family living in Union County GA included boys named Lafayette and John. These Kellys appear to have arrived from North Carolina about 1839. Since Lafayette's age is three, it's hard to imagine a boy well under twenty serving as a practicing physician during the war.
Where did the Kellys lived in 1860? Surely there's something more definite to go on. We're close to empty on this tank of gas.
The Kellys who served in the 4th Alabama Cavalry were from Madison County AL. The Lafayette Kelly mentioned was captured and died in Federal prison during the war. Many of the others are buried in Kelly Cemetery in Madison County AL.