All but one of the census records from 1880 to 1920 list him and his family as black. This might explain why he is not on any antebellum census.