Joan,

The Army of the Potomac did winter camp in Culpepper and the outlying areas from January (some accounts are as early as November)to May 1864 following the Mine Run Campaign. First Corps, Second Division Second Brigade was initially detailed on Provost Duty for Culpepper in the Jan-Feb timeframe. There was no POW camp there however. The circumstances of a Confederate Missouri Cavalryman finding his way to Culpepper would have to be extremely strange.

Below is the Wikipedia listing for all Union POW Camps;
Union camps
Alton Prison - Alton, Illinois [2]
Camp Butler - Springfield, Illinois [3]
Camp Chase Prison - Columbus, Ohio [4]
Camp Chemung - New York [5]
Camp Curtin - Harrisburg, Pennsylvania
Camp Douglas - Chicago, Illinois [6]
Camp Randall - Madison, Wisconsin [7]
Elmira Prison - Elmira, New York [8]
Fort Delaware - Pea Patch Island, Delaware [9]
Fort Jefferson - Key West, Florida [10]
Fort McHenry - Baltimore, Maryland [11]
Fort Warren - Boston, Massachusetts [12]
Gratiot Street Prison - St. Louis, Missouri [13]
Johnson's Island - Lake Erie, Sandusky, Ohio [14]
Ohio Penitentiary - Columbus, Ohio [15]
Old Capitol Prison - Washington, DC [16]
Point Lookout - Saint Mary's County, Maryland [17]
Rock Island Prison - Rock Island, Illinois - a government owned island in the Mississippi.

Many of these are closer to Missouri. Did Ellis perhaps reenlist and serve in a second unit? (Not uncommon in the eastern theater with three month regiments)
Good luck//Mike