There are 15 James Andersons listed on Louisiana rosters. I looked at only those enlisting in 1862 or later, whose age was not listed, and survived the war2. I assumed he didn't enlist in 1861, instead of being conscripted in 1862 or later.
Hope this might narrow the search.
This resulted in men serving in these units:
"I" Co. LA 27th Infantry
"B" Co. LA 11th Battn Infantry
"K" Co. LA Crescent Infantry
"C" Co. LA 1st Cavalry
LA 3rd Light Artillery James M. resident of Morehouse Parish
"C" Co. LA 12th Infantry
......
James B. Anderson, Private, Company C, 1st Regiment Louisiana Cavalry, captured near Maretta June 23, 1864, forwarded to Louisville, Ky. July 12, 1864, no other records
...
James Anderson, age 25, Private, Company B, 11th Battalion Louisiana Troops, enlisted May 6, 1862 at Natchitoches
...
James M. Anderson, Private, Company E, 12th Regiment Louisian Infantry, enlisted August 29, 1861 at Union City, Tennessee, transferred to Co. C, 17th Louisiana Volunteers, June 24, 1862, requested transfer "on account of all the relations I have in the army being in Capt. Taylor's Company"
...
James M. Anderson, Private, (2nd) Company C, 17th Regiment Louisiana Infantry, captured at Vicksburg, Miss. July 4, 1863, Sergeant Major, paroled July 10, 1863, no other records
...
James Anderson, Private, Company I, 27th Regiment Louisiana Infantry, died June 10, 1864 at General Hospital, Greenwood, Louisiana
...
J. J. Anderson, Private, Ruggles Guards,* appears on a list of members of the organization named above on parade, November 23, 1861, at New Orleans November 23, 1861
* This organization subsequently became Company G, Crescent Regiment Louisiana Infantry. On November 3, 1863, the Crescent Regiment was consolidated with the 16th Battalion Louisiana Infantry (also known as the Confederate Guards Response Battalion) and the 11th Battalion Louisiana Infantry, and their new organization designated as the Consolidated Crescent Regiment Louisiana Infantry
...
James M. Anderson, resident of Morehouse, Louisiana, Private, 3rd Battery (Benton's) Louisiana Light Artillery, paroled at Monroe, Louisiana, June 9, 1865, no other records
M320: Compiled Service Records of Confederate Soldiers Who Served in Organizations from the State of Louisiana