If you look at the record of Andrew Jackson Gardner (sometimes written J.A. Gardner), he was surrendered following the Battle of Ft. Henry (while in Wellers/Taylors Artillery Company), then took the Oath at Alton, IL in March 1862. Subsequently, he is in the 10th TN Cavalry through at least May 1863. The record ends thereafter. Of course, many men took the Oath, got out of POW camp and joined other units. Many men, no doubt, joined partisan units. That is probably what A.J. Gardner did after Alton and sometime in 1863.
So, when does taking the Oath become dishonorable?