I agree that there were abolitionists in portions of the North that frustrated Southerners quite a bit. But, abolitionists were a very distinct minority and did not represent the whole "North" (although, many Southerners perceived this to be the case).

I also have to agree that much of the Free Soil interests of northwesterners included a strong dislike for blacks. As you say, many white northerners wanted nothing to do with slavery OR blacks in their neighborhood. And this did place many Southerners in a complicated situation. Nonetheless, the actual institution of slavery was at the heart of the matter, as a property right (and Constitutional right) as well as an economic, political and social interest. Its future was at stake.