I've read numerous references to body armor in diaries or journals that have been published. In one account, a suttler proferred his wares and a man asked to verify it's ability to stop a bullet. A test was conducted in camp and it was found not to work.
Body armor was only worn by some during the first year of the war. Afterwards, it was discarded as useless and heavy (especially on the march). While I don't recall which book, I do recall one "save" by use of bodyarmor. Then again, we always hear about the belt plate, bible or something that stopped a bullet. Just read of an incident during the Revolutionary War where one fellow had a bullet flattened beneath the skin of his forehead. The surgeon "felt" it and carved it out. When it comes to ballistic performance, there are no guarentees and the body armor of the 19th century certainly isn't like today's kevlar or spectra shield.
Oh, the National Civil War Soldier Museum in Pamplin Park, VA has an example or two on display.