I believe that Buchanan's people were afraid that if Charleston and South Carolina were to be "let go", the first thing they would do would to make trade treaties with Europe. This would threaten a collision between the North and those countries in Europe who would come to South Carolina's aid to protect access to South Carolina's products. There was much talk of making Charleston a "port of entry" to Europe. This would put Charleston in direct competition with New York, although it could not out perform New York or Boston for many years if at all.
Lincoln had no real designs on holding on the Fort Sumter, his mind was on holding on to the South and how to get the South to act militarily in order to accomplish this. This is evident in how he planned to resupply the fort. Not with food but with arms and men; and then proceded to make it impossible for the plan to succeed and make it look like the South started the war.
South Carolina's problem was not with the forts as weapons of war but the fact they were occupied by foreign troops. If you will notice the Confederates in their negotiations with Anderson never demanded he surrender his command only that he evacuate the fort. Even after they fired upon him and he relented they never wanted to hold onto his command, they wanted his command on a ship northward bound. I always thought if it were possible the Charleston people would have let Anderson have his fort as long as he could take every brick of it with him north.