I'm sorry to say that I know nothing about the Virginia Browns. I just became aware of the family's connection to Gen Lee when my sister found a copy of the letter you mentioned.
My father was born in Forsyth, Ga to Samuel Bradford Brown and Undine Cater Brown. They moved shortly after his birth to West Palm Beach, FL.
The Cater family had a reunion in Atlanta many years ago, but I've never been to a Brown family reunion. My grandfathers sisters, Gertrude Brown and Susan Lara Brown were the family historians. I should have paid more attention.