The original purpose of Whigs like Henry Clay and Abraham Lincoln was two-fold: free the slaves and send them home, or elsewhere. Before the Civil War, the great majority of Republicans like Lincoln said many times that they had no intention of granting rights to black men that would make them equals of whites in American society. They wanted to fee the slaves, and if they could not all be sent overseas, have them placed on a reservation in Texas, Florida or Central America.

As the Civil War ground on and on, Republicans established the right of black men to serve in the U.S. army and began to envision them as a means to counterbalancing and punishing disloyal Southerners in the Confederate States. Those who supported the rebellion could be disenfranchised, and small numbers of loyal whites and "freedmen" (newly endowed with rights as citizens) could control those states for the Republican Party.

You might note that a lengthy struggle took place to apply the XIII, XIV and XV ammendments to the states. As late as the mid-twentieth century, a variety of states contested appliation of the "Bill of Rights" to internal state issues, and (if I'm not mistaken) the same is true of the Reconsruction ammendments. In other words, segregation or exclusion of people based on race - African, Asian or otherwise -- was allowed under state laws throughout the United States.