James Polk Fancher served in Co. E, 1st (Stirman's) Battalion Arkansas Cavalry. He enlisted at Washington Barracks (south of present-day Harrison) on January 1, 1862, and served for a time as battalion commissary sergeant. He was surrendered with his unit at Vicksburg, Mississippi, when that city capitulated on July 4, 1863. He was paroled and returned to Arkansas with his command to await formal exchange.

Stirman's battalion was dismounted and served as infantry when the Army of the West went to Mississippi, and at one point was increased to a regiment of sharpshooters. After Vicksburg, it was remounted in the Trans-Mississippi Department.