After the fall of Vicksburg, most of Company I of the 4th Mo Inf, deserted and went home rather than remain in a POW camp until the exchange could be accomplished. Why, because the situation at home was untenable, their families were in distress and needed protection. The Missouri Militia and Union Army were stripping the country side of everything needed for them to survive. Sometime in 1863 many of the 8th Inf came back to Missouri and joined Coleman. Capt. William Riley Skiles of the 8th became Major William Riley Skiles of Coleman's Regiment. All of these men were considered "bushwhackers" by the feds. Read the OR.
I believe the sterotypical dirt poor Missouri bushwhacker to be, in a large part, a myth. At least in my neck of the woods in south Missouri. They were not rich when compared to the blue bloods in little dixie, but comparing them to their neighbors, the southern supporters were among the top percentile in wealth when you look at the 1860 census in any of the southern counties.
The same types of illegal activities occurred by men on both sides. The south considered many Union officers and troops to be outlaws and if the south had won that is what we would think today. But, they didn't and as you said, the winner writes the history.
So, does that make it right for you to assume that just because a black man is riding with a bushwhacker outfit that he must have been just a "vicious criminal to begin with". You think about it and you decide.