Custer made bad decisions all throught that engagement such as splitting his command, and under estimating his enemy and their ability.
But he had twice the number of men that the British had at Rouke's Drift (Zulu). And had he formed square he would have force the attackers to consentrated their formation more and would have presented as more compact target for Custer's Springfield Trapdoor carbines.
It may not have changed the eventual outcome. But it probably would have made it more costly for the attackers. And may have caused them to break off their attacks before he was completely wiped out.
We'll never know. Ammunition supply would have been another factor. Cavalry troopers didn't usually carry a whole lot.