Using large language models as judges has become a common way to evaluate other models, often by pooling several judge decisions into a majority vote. But a new preprint argues that this panel-style aggregation does not guarantee that any single evaluation is trustworthy. The authors point out that even when the overall vote looks decisive, individual items can still be judged incorrectly.
To address this, they propose a structured method for examining the reliability of each evaluation instance rather than relying only on the final vote. The approach appears designed to flag cases where the judge panel may be confidently wrong, offering a more granular view of evaluation quality.
The paper does not dismiss LLM-as-a-judge entirely, but it cautions that aggregated scores can hide per-item failures. For researchers using these judges, the implication is that reliability should be checked at the instance level, not just at the level of the final verdict.