Men who simply wanted to be left alone probably outnumbered all the other factions. That was also true during the American Revolution.
Regarding Texas U.S. volunteers, two regiments of cavalry were recruited, but had to be consolidated by November 1864.