I did some research and found the following information. Perhaps the Confederate provost guards did not have as much notoriety as the Union guards.

On June 5, 1862, the Department of Northern Virginia put out a general order directing that provost guards be chosen for their reliability and efficiency. This order directed that each divisional guard would consist of one officer, one noncommissioned officer, and ten men from each regiment in the division. These men would be answerable to the division provost marshal (OR Vol. XI, Pt. 3:576-577).