Seems I recall reading that a "handful" of Knights "followers" soon left him, made it to Union lines (Ship Island maybe), and officially joined the army.