There was a James Madison Scott listed in "Souther Loyalists in the Civil War" but I am not able to access that information.
James Madison Scott
Born: 12 SEP 1815
Died: 24 APR 1899
Reidville, NC
Spouse
Cynthia E. Cobb 1822–1897
Children
Rebecca C. Scott 1840–
Hannah Scott 1842–
Adaline Scott 1845–
John W. Scott 1849–
James Jefferson Scott 1851–
Cynthia E. Scott 1853–
Elizabeth Scott 1857–
Lucy Ann Scott 1858–
Martha W. Scott 1861–\
Thomas W. Scott 1863–
Ann R. Scott 1865–
Ed & Rhonda Brewer's Family File at Ancestry.com
http://www.rootsweb.com/~nccaswel/
Caswell Rifles, Co D, 1st NC Inf Regt
Caswell Boys, Co H, 6th NC Inf Regt
Caswell Rangers, Co C, 3rd NC Cav Regt
6th Regiment, North Carolina Infantry
6th Infantry Regiment State Troops was organized at Halifax, North Carolina, in July, 1861. Its companies were recruited in the counties of Cumberland, Gates, Johnston, Graven, Rowan, Bertie, Wilson, and Caswell. Ordered to Virginia, the regiment reached Manassas on July 19 and fought in the battle under General Longstreet. In April, 1862, it had 460 effectives and during the war was brigaded under Generals Early, Garland, Iverson, R.D. Johnston. It participated in the campaigns of the Army of Northern Virginia from Williamsburg to Cold Harbor, then was involved in Early's operations in the Shenandoah Valley and the Appomattox Campaign. It had 180 men in action at Seven Pines, lost 10 killed, 22 wounded, and 4 missing during the Seven Days' Battles, and had 4 killed and 37 wounded at Chancellorsville. The unit took 473 men to Gettysburg, losing more than half, and reported 16 disabled at Bristoe and 3 at Mine Run. It surrendered with 7 officers and 76 men of which 48 were armed. The field officers were Colonels Thomas M. Garrett and Duncan K. McRae; and Lieutenant Colonels John C. Badham, William J. Hill, Joseph P. Jones, John W. Lea, and Peter J. Sinclair.
Soldier Name Company Rank In Rank Out
Scott, James C.
Co B Private Private Co B - Madison Rangers
Scott, Thomas
Co K Private Private Co K - Columbus Rifles
Scott, William R.
Co K Private Private
1st Regiment, North Carolina Infantry
1st Infantry Regiment State Troops was organized at the race track near Warrenton, North Carolina, during the spring of 1861. Its members were recruited in the counties of Chowan, Wilkes, New Hanover, Orange, Lincoln, Hertford, Northampton, Washington, Martin, Wake, and Halifax. In July it was mustered into Confederate service with more than 1,500 officers and men and ordered to Virginia. The regiment was brigaded under General Ripley, Colston, Steuart, and Cox. It participated in the campaigns of the army from the Seven Days' Battles to Cold Harbor, was with Early in the Shenandoah Valley, and shared in the Appomattox operations. This unit reported 142 casualties at Mechanicsville, 75 at Malvern Hill, 160 at South Mountain and Sharpsburg, and 15 at Fredericksburg. It lost 34 killed and 83 wounded at Chancellorsville and forty percent of the 377 at Gettysburg. It surrendered 10 officers and 61 men in April, 1865. The field officers were Colonels Hamilton A. Brown, John A. McDowell, and Montfort S. Stokes; Lieutenant Colonels Jarrett N. Harrell and Matthew W. Ransom; and Majors James S. Hines, L.C. Latham, and Tristim L. Skinner.
Soldier Name Company Rank In Rank Out
Scott, A.H.
Co G Private Private Co G - Burke Rifles
Scott, Albert H.
Co G Private Private
Scott, Edward M.
Co D Captain Captain Co D - Caswell Rifles
Scott, Harrison
Co D Private Private
Scott, J.H.
Co G Private Private
Scott, James G.
Co D 2 Lieutenant 2 Lieutenant
Scott, Obadiah R.
Co C Corporal 2 Lieutenant Co C - None Stated - person listed here is probably the same person listed below - twice
Scott, Obed R.
Co C Corporal 2 Lieutenant
Scott, R.S.
Co C Corporal 2 Lieutenant
Scott, Walter
Co D Private Private
3rd Regiment, North Carolina Cavalry
"Units of the Confederate States Army" by Joseph H. Crute, Jr. contains no history for this unit.
Soldier Name Company Rank In Rank Out
Scott, Benjamin
Co H Private Private Co H - Humphry Troop
Scott, J.S.
Co F Private Private Co F - Davis Dragoons
Scott, James G.
Co B Private 1 Sergeant Co B - Gatlin Dragoons
Scott, John W.
Co C Private Private Co C - Caswell Rangers - could this be the son of James Madison Scott?
Scott, L.M.
Co F Private Private
Scott, Lucius C.
Co F Private Private
Scott, R.S.
Co B Private Private
Maybe this will help.