You may be referring to the college at Chapel Hill in the southwest corner of Lafayette County between Columbus in Johnson County and the Sni-A-Bar Hills. The structure was burned some time during the Civil War. I seem to recall that the ruins were still visible back in the 1970s, but I am not sure.

Bruce Nichols