Mary Todd Lincoln's family had "house Servants".

"Despite growing up in the south with household slaves, Mary very early on grew to abhor slavery."

http://www.lkwdpl.org/wihohio/linc-mar.htm

I read that Lincoln's law partner claimed to have changed Lincoln;s views on slavery, but couldn't find anything concrete on it.