This is most likely your man and his family:

Name: Joel Young
Age: 9
Birth Year: abt 1841
Birthplace: Alabama
Home in 1850: Beat 12, Randolph, Alabama, USA
Gender: Male
Family Number: 1254
Household Members:
Name Age
William Young 40
Ellen Young 31
Andrew Young 13
James Young 11
Joel Young 9
Elijah Young 7
Augusta Young 4
Elizabeth Young 1

Could not find him on the 1860 census.

Joel Young - also shown as and filed under Jouel Young, enlisted Sept. 14, 1861, at Montgomery. Last dated record shows that he received pay on Sept. 1, 1863. Final record is of Effects of Deceased Soldiers, turned over to Quartermasters, C.S.A. (No month or day listed - only the year 1864) amount $10.00.

http://www.archives.alabama.gov/referenc/alamilor/17thinf.html