That leaves 3/4's of the population not interested in protecting slavery doesn't it? The problem with the number of slave holders on the census record is it doesn't separate multiple owners in one family or individual owners in other districts. Many well-off plantation owners owned slaves in different counties, cities and states and were counted multiple times. Also, many slave owners lived outside the state like New York or Massachusetts, so how were they counted?
I'm sure you can find papers filled with debate that they were fighting for slavery, but my point is, you rarely see or hear from historians references to those Southern voices calling for emmancipation to win the war. They exist. By 1864 the Army of Northern Virginia was ordered to poll the troops on this question, overwhelming they approved the measure to allow the arming of slaves.