I think you're right Dennis. I learned in Genealogy diggings that what you heard from a family member, even if it's written in a letter may be the exact opposite of something in the OR's or other documents.
I have wondered if my GGGrand might have gone Yankee had they not turned him loose thinking he was harmless since they'd shot off his arm. He was a POW for a few months and I believe some others with him did join the other side just to get out of that situation.
He's the one I'm trying to figure out how he was safe, then just had to go back for his horse, and that's when he was shot and captured. Now, he was infantry? A horse? But, he went back to the 32nd Alabama after promising to go home since he was 'crippled' and became a courier -- on a horse! I'm hoping I don't come on some other document that says it was a Yankee horse. Especially after there's a nice little Confederate Flag and Soldier at his grave with a plaque telling of his 'exploits'. Nothing's said about the still he ran after the war though. Imagine that!