You may have gotten the name off a little bit.

If you go to this Walker County 1850 census page you'll see this.
http://www.rootsweb.com/~cenfiles/ga/walker/1850/Pg0408b.txt

Kelly Phoebe C....50...F......N C
Kelly T L........24...M......N C......Physician
Kelly D U........20...F......N C
Kelly F C........16...F......N C
Kelly L A........13...F......N C
Kelly Eugenia....10...F......N C
Kelly John A......8...M......N C

Eugenia I would guess is your GGGrandmother John Kelly has a middle initial A. And Lafayette must have been his middle initial. His first name starts with a T and he is listed as a physician.