Oliver John Semmes was born 29 Aug. 1839; died 19 Jan. 1918 in Mobile, Alabama; parents, Raphael Semmes (1807-1877) and Ann Elizabeth Spencer (1819-1892); marriage, Amante Gaines (1849-1888) 17 Dec. 1873, Mobile, Ala.; children, Amante Elctra Semmes (1874-1951), Oliver John Semmes (1876-1957) and Raphael Owen Semmes (1879-1910).
He is listed in the 1860 census in New York, where he was attending West Point and on the 1880 census was living in Mobile, Alabama.
I found a really interesting sketch on him at the following: