I've read several accounts over the years. The eighth paragraph down in this article talks about it. Like most if not all, the Unionists in east Tennessee, Andy was pro-slavery and as racist as they come. Sometime, find online the account of his confrontation in the White House with Fredrick Douglass.

https://opinionator.blogs.nytimes.com/2013/08/09/when-andrew-johnson-freed-his-slaves/