Loyalty among body servants should have been stronger than among much larger groups of African-Americans in the Confederate army. The reason isn't difficult to fathom - in many cases, these men had grown up together and were life-long companions. Yet we know that some deserted because they crossed enemy lines and provided information to Federal officers. In many reports the man's owner is identified to confirm the authenticity of the information the slave delivered.
Another important point - someone had to feed and provide for body servants. Normally only officers could do that. Confederates who had body servants with them for long period of the war eventually had to send them home because rations were cut and cut again by the Confederate Commissary Department. They simply had nothing to share. That will explan why only a small number of black men were still with Lee's army at Appomattox.