Which isn't denying that your relatives might have been victims of Union soldiers, but there were no 'invading' Union armies in Kentucky.
Now, if you're asking what forces on either side might have been in the area at the time, then one of the historians here could probably give you a place to look. Have you checked with the local history society? What documentation do you already have that you can use as a starting point?
Also, don't overlook guerilla activity - what you describe sounds more like guerillas to me. Many guerillas were active in the area, and many also crossed over from Missouri.