According to his enlistment papers, this Edwin J Holley was 20 years old when he enlisted. Perhaps he was a son of your Edwin J Holley? There was a Haster (or Hasten) Holley, who enlisted on the same day in the same company as Edwin, who was 29; I would guess this was probably a brother? Both men were born in South Carolina. Here are there records from the Mormon Genealogy records and from Booth's records. Their service records can be ordered from this website by clicking on the link at the top of the page.

EDWIN J. HOLLEY, Pvt., Co. B, enlisted 6 May 1862 at Natchitoches, LA, born Kershaw Dist. SC, age 20, dark eyes, dark hair, dark complexion, occupation farmer & was 5’ 6 ¾” tall. Captured on board Queen of the West 14 Apr 1863 at Grand Lake,
LA.

HASTEN C. HOLLEY, Pvt., Co. B, enlisted 6 May 1862 at Natchitoches, LA, born Chesterfield Dist. SC, age 29, blue eyes, dark hair, light complexion, occupation mechanic & was 6’ tall.

Holley, E. J. (also Ahlley, E. J.), Pvt. Co. B, 11 Battn. La. Inf. En. May 6, 1862, Natchitoches. Rolls from May, 1862, to Feb., 1863, Present. Roll for March and April, 1863, Captured on board Queen of the West, April 14, 1863. Federal Rolls of Prisoners of War, Captured at Grand Lake, La., April 14, 1863. Paroled May 11, 1863, at Pt. Hudson. Also on Rolls of Co. K, Cons. Cres. Regt. La. Inf. Pvt. Roll for Jan. and Feb., 1864, Present. Return dated Camp Buckner, Dec. 31, 1864, Died of wounds Recd. Mansfield, La., April 17, 1864.

Holley, H. C. (also Halley, H. C), Pvt. Co. B, 11th Battn. La. Inf. En. May 6, 1862,
Natchitoches. Rolls from May, 1862, to Feb., 1863, Present. Roll for March and April, 1863, Absent on detached service. Det. March 1, 1863, as carpenter at Ft. De Russy. Also on Rolls of Co. K, Cons. Cres, Regt. La. Inf. Pvt. Roll for Jan. and Feb., 1864 (only Roll on file), En. May 6, 1862. Natchitoches, La. Absent. Detailed as carpenter at Alexandria, La., Aug. 20, 1864, order of Gen. Taylor.