That is because ever since President U.S. Grant (Hero of the Union) vetoed the Civil Rights Act of 1872, it has been a "black" issue. The United States Supreme Court ruling in Plessey v Ferguson case made it a segregation issue. The United States Supreme Court Ruling in Brown v the Kansas City Board of Education case made it a Civil Rights issue. And affirmative Action Legislation made it a Racial issue. Lastly, the United States of America wasn't the first, or the only ones to do it. We are just the only ones who have so far fought a war that killed maybe 2.5 million of our soldiers and citizens over the issue. Which even then wasn't the real cause of that War. But in point of fact was a sham issue to justify that war and a political ploy which had to be "honored" after the fact.
I guess it is the hypocricy of the claim that we freed the slave for any benevolent reasons.