The slaves. Sometime you should pick up some books of interviews. You would be very surprised.
The owners did not care for him because he would not allow the families to be broken up and sold to different owners. He also would not sell to anyone with a reputation of abusing slaves.