Yes you are correct.

Ironically we can see the danger of the Constitution's lack of perpetual union during the War of 1812, when New England was on the verge of seceding and its lack of real military support to the rest of the nation.

Had Andrew Jackson failed at New Orleans, New England would be part of Canada and the Louisana purchase would belong to Mexico via Great Britian to Spain.

There were fears in the 1860s but no legal external enemy existed. France in Mexico and Great Britain in Canada were a worry and so much so that Russia sent its navy to Northern ports as a show of solidarity against Western Europe during the Civil War. But this fear did not fully materialize until the war started.
_______________________
David Upton