I found it! It wasn't a letter to lincoln; it was an essay in Douglass' monthly newspaper, September, 1861, called "Fighting Rebels With Only One Hand". You can read the whole speech at: http://www.frederickdouglass.org/speeches/index.html

The paragraph begins: "It is now pretty well established,"; some may take this as meaning Douglass had no real evidence. After the part usually quoted, he goes on "There were such soldiers at Manassas, and they are probably there still. There is a Negro in the army as well as in the fence, and our Government is likely to find it out before the war comes to an end. That the Negroes are numerous in the rebel army, and do for that army its heaviest work, is beyond question." He then outlines several tasks performed by Blacks for the army.

THen: "Rising above vulgar prejudice, the slaveholding rebel accepts the aid of the black man as readily as that of any other. If a bad cause can do this, why should a good cause be less wisely conducted?"

Douglass is using what he appears to accept as true to argue that the North should do lkewise. Stan