Julie --
Find a copy of Bluegrass Confederate: The Headquarters Diary of Edward O. Guerrant, William C. Davis and Meredith L. Swentor, eds., LSU Press, 1999.
Clay is mentioned 20+ times. The last of these, dated Apr. 23, 1864, mentions "Lt. Col. E. F. Clay, in com'd of his own & Johnson's & Jessee's batt'ns, went down to Paintville...." He continues to describe how Clay was wounded and captured when his command was surprised in camp.
Clay has service files with the 1st and 3rd Kentucky Mounted Rifles. These are battalions, not batteries. You can order copies of these here --