An old timer quite a few years ago mentioned this phrase to me.It meant giving up like a cowardly yellow dog would do.
Signing an Oath of Allegiance was a bitter pill to swallow for most of the men. Swallowing the " yellow dog" meant that the men were resigning themselves to giving up the fight. To Yellow ( or cowardly) to keep up the fight. Although this was really not the case for most men. They just hated to do it. A cowardly dog or Yellow dog that wouldn't fight wasn't worth owning.

Hope this helps.