As we have discussed earlier the arming of the Blacks in the south was a very touchy subject. The main concern was the possibility of a slave revolt was very much a concern which was fostered by the John Brown Harper's Ferry Raid in 1859 which resulted in the quick rise in the formation of local Militia Companies all throught the south in 1860. This was one of the causes of uneasiness when Federal troops were sent to occupy previously unoccupied arsenal throughout the South in 1860. The south saw this as a possible move of Northern abolishionist to arm the southern slaves then.
When the Union Generals like Ben Butler started to arm black troops like the Louisiana Native Guards and later the U.S. Colored Troops was seen as an extension of that fear of Northern abolishionist arming the southern Blacks to fight against the Southerners. Therefore the Southern people were reluctant to arm large numbers of Blacks for that reason that they may revolt. This does not mean the the Individual servant who was trusted and had demonstrated their loyality like Holt Collier were not permitted to have firearms.
To me this was one of the BIG reasons for the changing of attitudes of southern white against the Blacks as a whole.