After all the great help from you folks, I think I have found my ancestors Edwin Holley and his son John Holley, both were in the Union Army in Louisiana. Not sure who the confederate Edwin J. Holley was but he's not my ancestor. Below is what I've found.

Edwin Holley - Capt. Peter's Co. 1st regiment L.A. Scouts He died at the general Hospital in Alexandria on April 19, 1864 from Typhoid fever.

John Holley - Co. B 2nd Regiment L.A. Mounted Infantry - He enlisted on April 18, 1864 at Alexandria. He stated he was 23 years old and born in Monroe County, Alabama. In August of 1864 he was convicted in civil court of having some stolen cloth that he had bought while on a furlough. His sentence was 10 years and he was writing appeals to general Banks trying to get released. I think he must have got released because he was back in the regiment in early 1865 and mustered out on Dec. 18, 1865. He was transferred to the 1st Regiment Calvary in early 1865.

Any service records for Edwin Holley would be great, all I could find was the info when he died from the hospital. I also would like to see if his son John Holley drew a pension. He moved to Texas by 1870 so it may have been there.

Thanks, Mike