Bryan,
I had to go back through my notes but here's something that may help. According to In The Saddle With The Texans
(Parson's 12th Texas Cavalry Order Book) on page 45 it states, "Hd. Qrs. Tex. Cav. Brigade Camp Cache River Aug 23d 1862"..."Capt. W.G. Veal will assume cmmd. of the hospital camps at Brownsville..."
If memory serves me correctly there are several references to hospital camps at Brownsville in the book.
Confederates seldom if ever camped close to any population center as you say so these camps would probably be located two to three miles from town. Brownsville Station was about two miles south of Brownsville.
I hope this has helped rather than muddy the waters.