From a "History of Discipline in the House," John B. Clark of Missouri was expelled for "Disloyalty to the Union - taking up arms against the United States."

See: http://reporting.sunlightfoundation.com/2010/history-discipline-house/