Confederate mounted troops were usually short of arms throughout the war. Surprisingly many reports show some cavalrymen without any arms whatsoever. For example in March 1863 General Polk reported that Russell's 4th Alabama Cavalry shouldn't be on picket duty because a majority of enlisted men had no arms. The greater number of Confederate cavalrymen who carried weapons had long arms. These were all but useless in the saddle.

Local defense companies like Pemberton's would have been the last to receive any kind of weapons from an armory.
You might check Columbus GA newspapers to see what if anything appeared in print about this company.