After investing a couple of hours of census research, I don't see any possible connection. In 1880 Miles I. Taylor and his family lived in Montague County TX. He doesn't have a son named Charles or Charley.

Dexter B. Taylor, lived in Lamar County AL in 1930, wife's name Ola. He's in Lamar County in 1920, age 13, living with parents Charley, age 55, and Frances, both born AL. In 1910 he's still in Lamar County, age 3, parents Charles W. and Dovie F, both born in AL.

In 1900 Charley H. Taylor lives in Lamar County AL. He's born AL Apr. 1862, (parents born SC and NC) wife Debbyann F., born Mar. 1873. They married eight years earlier. Identifications for him on the 1880 census are unclear, but he may be listed as Charles Taylor, hireling (wage laborer), age 21, working in Fayette County. He also could be the Charles S. Taylor in the household of Jonathan Taylor, born TN 1826, wife Sarah A., son Charles S 14 AL. That family appears on the 1870 census of Fayette County AL, Charles being in the household as C. S. Taylor.

Wish I could help you further with this.