From my research for many years now is that men in North Georgia that were "picked up" during May 1864 until the end of the war were deserters. There were over 10,000 men at Kingston, Georgia that surrendered. That is a full division of folks that could have been used in either the ANV or AoT.
If they went home on furlough, they stayed well past the time period a soldier would have been allotted for leave. The leadership would not approve a soldier going on leave behind enemy lines to go visit family. If I am not mistaken, leaves were suspended in either late March or April of 1864. It would have been equivalent to the modern day "Stop Loss."
The leadership did not have a vast "spy network" and continually sent "Scouts" into the North Georgia mountains once the campaign kicked off on May 5, 1864. Recruiting duty would only be left to officers but rare at this point in the war.
If a soldier was "sick" at this point in the war, then they sent them somewhere in the General Hospital system. If a soldier was wounded and it would take him more than 60 days to recuperate then it was likely they could not travel and certainly could not evade capture to get into the mountains of North Georgia. They sent them to convalescent camps further south. These hospitals were outside the General Hospital system overseen by the Director of the Army of Tennessee hospitals, Samuel H. Stout.
Sorry, I have seen descendants or amateur historians hemming and hawing for many years now trying to talk around, rationalize, or excuse the obvious. Sure, there was some that fell into those categories but often, in my experience, they deserted.
Please let me know if you have any questions.
Gerald D. Hodge, Jr.
Lieutenant Colonel, U.S. Army (Retired)
M.A. Military History - Civil War Concentration
Research - Preservation
Historian: 39th Georgia Volunteer Infantry Regiment