Fielding was not elected by people who hated him. He was elected by carpet baggers and a minority of Unionists. Most of the constituents were Confederate veterans and ineligible to vote.
p.s. He had very long leash for a "rogue" officer.