How about this scenario? Arizona, New Mexico and perhaps another territory stall about annexation due to concerns about the Lincoln Regime's centralized government and butchering of the 9th and 10th Amendments. Meanwhile, the C.S. is pre-occupied with re-building their economy and defense. The U.S. sends their army to invade these obstinate territories to change their minds about statehood. War erupts and the Californicators secede from the Union. Manifest Destiny implodes.

I agree with your opinion that Secession of the Southern States would not stand with the goal of Manifest Destiny. But, without the slavery issue, there could be some European intrigue in the future war(s).

And whatever happened would be better than what did happen IMHO.