It may be that he's on the 1870 census of Henderson Co. under a different first name. There are at least two men named Diffee or Diffey who are about the right age.