Elk v. Wilkins (1884) confirmed that Native Americans could not become American citizens. That restriction remained until the Indian Citizenship Act of 1924.
http://en.wikipedia.org/wiki/Elk_v._Wilkins
Regardless of the abolition of slavery by the XIII Ammendment, the limited rights granted to resident aliens made it possible for American citizens to treat them as slaves. That was also true for Asian and other non-white immigrants.