Hopefully someone knows some real information on your James Warden. However adding my two cents for what it is worth, and speaking from experience, if you have reason to believe that he did not desert, he probably did not.
My ancestor was listed as deserting after the Battle of Shiloh. That man was at home healing after being shot and left with a bad leg fot the rest of his life.
He still got up and marched and fought all over the South being listed as dying at Atlanta. Since I did not provide the information at that Tennessee site, they will not take it down. He fathered at least 12 children after the war ended, including 8 by a woman in Arkansas. I have his pug nose and his birthmark so he did not die, nor did he desert.
If your James Warden deserted, a good question would be, where was his group during this time.
If it was Dec. of 1863 and if he was close to home, he may have just gone home. The whole thing is you need to figure out a time line. Or if he decided living was more important than "the cause," then you may wish to try and understand.
My ggg uncle did desert and went home to his wife and little girls. It was spring time and time to put in the crops. That fall he joined the Union army. He had three brothers that stuck with the Confederacy and two of those died, one right after Shiloh and the other in a Yankee Prison after being captured down around Natchez, I think it was.
Life was not easy no matter where you were during that time.
Good luck in your search.