Here are a couple of possibilities:
(Journal of William Meshack Abernathy, B/17 MS) Simon was together with him since the beginning of the war, through the surrender at Appomattox. Simon attended to Abernathy in a Union hospital in Philadelphia after he had been wounded at Gettysburg. Abernathy was carried back to the hospital on July 2 and the first man to find him was his colored servant, Simon.
(Histories of the Several Regiments … article by Wharton J. Green?; and Wharton Jackson Green, Recollections and Reflections - Wharton was a volunteer aide to Brig. Gen. Junius Daniel) 4 July. Wm. R. Bond of Gen. Daniel’s staff, likewise wounded and myself were assigned to a one horse wagon driven by Guilford [black servant lad of Wharton Jackson Green]. At Monterrey pass, Federal cavalry surrounded them. Guilford said, “Don’t shoot, gentlemen, for God’s sake, don’t shoot. We surrender. We are prisoners.” Guilford wound up with Green at the General Hospital in Frederick, Maryland, and was likely sent on to a Federal prison.
At other prisons:
(Fort Delaware?) Daniel Burton, body servant of George W. Finley of 56 VA, was captured with Finley at Gettysburg and ended up at Fort Delaware with him. Apparently Burton stayed at Fort Delaware, possibly working there as an orderly. (Finley, 56 VA, Civil War Prisons Board)
(13th Virginia Cavalry, article on Richard (Dick) Poplar) At beginning of war, H/13 VA Cav “Sussex Dragoons” was a wealthy organization and each member of the company had his own servant along with him. Richard Poplar (colored) was a cook. He was captured on retreat and confined at Fort Delaware for five months, then Point Lookout for 14 months, refusing the Oath of Allegiance. Per CSR, captured July 5 at Greencastle, PA.