Do they qualify as being loyal?
A somewhat similiar situation took place in the fall of 1864. Forrest captured several hundred black soldiers (mostly 111th U.S.C.T.) who were sent to work on fortifications around Mobile. Bob Bradley at the ADAH has collected a considerable amount of material on the "Forrest Negroes."
I don't see your reference to "pound sand". Please explain.