David, there's an interesting account of Elisha Baxter in Goodspeed's, which expands on the point you raised --

He was a Whig in polities, and a strong adherent of that party until it disbanded in 1855. Mr. Baxter then attempted to co-operate with the Democratic party, but could not agree with them on the question of secession. As a Whig, he was elected and served as mayor of Batesville in 1853, and in 1854 was elected a member of the legislature from Independence county, which had not elected a Whig to any position for twenty years. In 1858 he was again elected to the legislature as a non-partisan, and in 1860 was defeated for prosecuting attorney of the Third judicial district by F. W. Desha. When the war came on he tried to be neutral and loyal to the government of the United States, so that when Curtis came into Batesville with 20,000 Federal troops, in the spring of 1862, his position enabled him to do a great deal of good for the citizens of Batesville; and during the two months that the place was occupied by the Federal army he was incessantly engaged in reclaiming property, collecting vouchers, and procuring the release of prisoners, without the hope or prospects of reward. At that time he believed his course would be appreciated, but Curtis had scarcely left when he was notified by some friendly Confederates that he could not safely remain in the country. Accordingly, he left on short notice, with but very little provision made for himself and family. Overtaking Curtis at Jacksonport, he was tendered the command of the First Arkansas Federal regiment, then just ready to be organized, but declined, and as he said to General Curtis: Not because I think you ought not to whip the rebellious, but because I feel that I, who am Southern born and raised, ought not to take arms against my neighbors and friends.

He did not get to see or hear from his family for almost a year, and in the spring of 1863 he was captured by a squad of Southern cavalry commanded by Col. Newton. On arriving at headquarters he received such courtesy from Col. Newton, and discovered in him such military genius, that afterward, when he became governor, and felt it his duty to appoint a major-general for active operations in the field, he did not hesitate to bestow the commission on Col. Newton, who had paroled him at Fredericktown, Missouri, with an escort of two men, and required him to report to Gen. Holmes at Little Rock. He had scarcely left Newton’s camp when he came in full view of the Federal army, in which one of his friends urged him to join them, but he replied that he had given his pledge of honor to report at Little Rock, which he did, and Gen. Holmes unceremoniously turned him over to the civil authorities, who assigned him to the Pulaski county jail to await an indictment for treason against the Confederate States. In due time the indictment was found, and he was arraigned before Judge Ringold, William M. Randolph acting as district attorney. The case was continued until the next term of court, and through the agency of some friends he managed to escape from jail, and after concealing himself for eighteen days near Little Rock, without any shelter and barely enough food to live, he succeeded in making his way into the Federal army, then at Little Rock, Gen. Steele having captured and occupied that place on September 10, 1863. In all this time he was abused, verbally and through the press, being branded as a coward by the True Democrat, of Little Rock, citing his refusal to take command of the First Arkansas Federal Regiment, when tendered him by Gen. Curtis, as a proof. Stung by these reflections, he made haste to apply to Gen. Steele for authority to recruit a regiment for the Federal service. Armed with such authority, he proceeded to Jacksonport and recruited the Fourth Arkansas Mounted Infantry, and reported to Gen. R. R. Livingston, at Batesville, where he commanded the post until the spring of 1864, when, under the organization of the Murphy or war government, he was elected a member of the supreme court. Under the constitution of 1864 he could not hold two offices at one time, so, after much hesitation, he resigned his command of the regiment, which devolved upon his brother as senior captain, as well as by order of Gen. Steele. Fourteen days after becoming a member of the supreme court of the State, he was elected, over his protest, to the senate of the United States for the long term. He repaired to Washington, taking his family with him for safety as far as Illinois, and presented his credentials, under the State government then existing in Arkansas, but was not permitted to qualify as senator. After the war was over he returned with his family to Batesville, and resumed the practice of law.