Half-brother
Henry Lee IV aka Black Horse Harry
1787-1838
Full-brothers
Algernon Sydnee Lee
1795-1796
Charles Carter Lee
1798-1871
Sydney Smith Lee
1802-1869
(father of Fitzhugh Lee)
Perhaps it was a brother-in-law, or a nephew.