Disagreement on slavery increased during the years leading up to the war. However, the vast majority of white Americans -- North and South -- didn't want civil rights granted to non-whites. This included American Indians, Mexicans, Asians and Africans, free or slave. For example, Abraham Lincoln opposed acquisition of more territory in the Southwest because it included people of "mixed blood". Also, state laws in states composing the old Northwest Territory excluded slavery, but forbade free people of color from making contracts, establishing residency, serving in the militia, sending their children to public school, holding civil office or voting.
While relatively large numbers of free blacks lived in cities from New York to New Orleans, Americans generally favored their removal. The Whig Party's long-term goal of colonization is well known. It's interesting to note that AFTER the war, General Forrest made speeches in favor bring more people from Africa to the South.