Fort Phantom Hill was established by the U.S. Army in 1851 and used until 1854 when it was abandoned even before the War Between the States began. A building of the fort was used as Station No. 54 on the Butterfield Overland Mail from 1858 until the war started and the Butterfield Overland Mail was closed in Texas and rerouted through Nebraska to California.

During the War Between the States, Fort Phantom Hill was a patrol camp for some of Capt. Henry S. Fossett's company of Lt. Col. Buck Barry's Frontier Battalion, C.S.A. Capt. Fossett's 160 man company was stationed at Camp Colorado in northern Coleman County for the protection of the frontier about 90 miles south of Camp Colorado. The Frontier Battalion was only used to protect the frontier from outlaws, deserters and Indians during the war. Capt.Fossetts company saw minor patrol action against roaming Indians and one major engagement against Kickapoo Indians at the disasterous battle of Dove Creek on Jan. 8, 1865, on the cattle range used by James Tankersley, a few miles west of present San Angelo, Texas.

The Frontier Battalion of the 2nd Frontier District saw no action with federal forces.