No one has mentioned it, but the concept of emancipation and arming black men were controversial issues in the North as well. The 128th Illinois was one of the recently recruited regiments which disbanded due to dissent over Lincoln's policies. Almost everyone in that day held strongly negative opinions about race, North and South. For example, many of Abraham Lincoln's speeches which reference people of other races contain what would be described today as objectionable language and ideas.
See, for example, Lincoln's speech at Carlinville IL, Aug. 31, 1858 --
For that matter, a deadly race riot in Lincoln's adopted home town was the incident behind formation of the NAACP.