Alan,
Generally true, although a few African-Americans did serve in the 1st Florida U.S. Cavalry. They are listed as such on the rolls and performed manual labor for the most part, but as opposed to Confederates in the area, they were at least identified as members of the unit.
Dale