Brian, the only additional information I have comes from an index of U.S. veteran's pension applications. There are entries for all four men. You can order a copy of the pension records from the National Archives and Records Administration.

Abraham Oliver applied for U.S. veteran's pension in 1881. According to the pension index, he died at Greenbrier, West Virginia, on September 11, 1908, and his widow (not named in the index) applied for a pension in 1911.

Henry Oliver applied for U.S. veteran's pension in 1882. According to the pension index, he died at Atlas, Arkansas, on December 7, 1930. No record of a widow's pension application.

Richmond Oliver died during the war, March 21, 1865, and his widow (not named in the index) applied for a pension in 1865.

William Oliver was killed during the war, March 26, 1864, and his mother (not named in the index) applied for a pension in 1864.