I just want to bring up something that seldom gets mentioned. While many Union troops, such as A.J. Smith's during the Red River Campaign of 1864, were known to be arsonist there were also Union troops who did not approve of these events and actually tried to stop them.
I feel there are two reasons for this. Southerners hate to admit not all Union soldiers were firebugs and northerners hate to admit decent troops from the north had to contend with these type of men in their own armies.
Most people don't realize that the town of Alexandria, Louisiana (near my home) was put to the torch. I suggest reading this article. Be sure to notice the conduct of the 113th New York...
http://www.libertychapelcemetery.org/files/a_burns.html