An arXiv preprint from Kozodoi, Afolabi, and Butler asks where an LLM should be made to differ when several answers are aggregated at test time. Standard self-consistency resamples the reasoning path from a fixed prompt; the authors instead compare test-time augmentations that modify the input itself, by rephrasing the question, perturbing its spelling, or transforming accompanying images.
Across six benchmarks, with the number of inference calls held constant, they found that the source of diversity matters more than its quantity. Rephrasing the question in a meaning-preserving way produced the largest accuracy improvement, +1.80 percentage points over a single call, and reached peak accuracy with the fewest samples. Character-level noise made responses disagree more often—37.8% versus 27.4%—but yielded smaller gains. Perturbing two modalities at once fell below the single-call baseline.
The authors argue that not all disagreement is useful: perturbations that preserve meaning create disagreement that aggregation can exploit, while corrupting perturbations create disagreement that is simply wrong. For a mid-tier model such as Claude Haiku, they conclude that spending an extra inference call on rephrasing the question is a better use than varying the reasoning path. The implementation is released with the paper.