You got that right. The point is there were evidently a whole lot of "white" Africans running around in the south. Remember that this, as in the Thomas Jefferson case, had been going on for 250 years before there ever was a Confederate States of America.
As you point out a person that was 1/32 African heritage is not going to display a whole lot of distinguishable African ethnic characteristics in appearance. I wonder how many "white" Confederate Soldiers, especially from states like Louisiana and Texas, were actually of unknown African heritage?