Political leaders who had fled from Richmond met with Forrest at Gainesville AL. These doubtless included Congressmen who debated the ridiculous Negro enlistment act of March 1865. They urged Forrest to take his men west of the Mississippi and continue the struggle there. Forrest replied, "Ya'll can do what you damn well please. I'm a'going home."

That quote appears in this book --
http://www.amazon.com/Battle-Start-Nathan-Bedford-Forrest/dp/0060168323