Hello. My own gg grandfather William Robinson Vaughan of Little Rock, Arkansas refused to sign a loyalty oath.
Since he refused, he never received payment for the cotton sitting on the docks that the Union army took when Little Rock fell in September of 1863. Therefore, he died flat broke in about 1877. The reason for not signing was that it "had cost him too much."
W. R. Vaughan had been part of a milita group that was formed in his area.
His nephew Frank Vaughan fought at Helena and was later a judge in Little Rock.
His own son died in some sort of tragic thing over a wagon load of salt during the Civil War.
Vaughan laid in an unmarked grave until 1997 when one of his many great great grandchildren, a nice lady from Texas, had his grave marked with a CSA marker given out by the U. S. government. The dedication ceremony was quite moving.
I always thought that was such a sad story. So not signing the loyalty oath could cost you plenty, and that would be enough reason for almost anyone to sign is my opinion, but I do applaud/admire W. R. Vaughan for being a man of convictions.