If we're talking about refugees and civilians, there are many indications that the Confederates viwed Southerners hostile to the Confederacy as guilty of treason.
ML