I can't swear to it but I think that quote is from Andrew Johnson. It is more in keeping with his political thinking. Brownlow and Johnson were like minded on many issues but the Parson despised democrats any and all.
In the middle of a canvas the Whig called Johnson a liar, a bastard and an athesist. That forced Johnson to have to journey to Raleigh to find proof of his legitimacy. When Johnson had the audacity to educate one of his daughters in Catholic school Brownlow had a coniption.