Label-free evaluation methods—such as self-consistency filtering, majority-vote aggregation, and agreement-based reliability scoring—share a common premise: when multiple outputs or judges agree, that agreement is a reliable signal of correctness. These approaches are attractive because they do not require ground-truth labels, but they rest on an implicit assumption that consistency can serve as a proxy for accuracy.
A new preprint challenges that assumption directly. The authors report a consistency-accuracy dissociation: judges can agree strongly and still be wrong. In other words, high agreement among model outputs or evaluators does not guarantee that the underlying answer is correct, and methods built on this premise may offer false confidence.
The abstract is truncated, so the full conditions of the dissociation are not available, but the core finding is clear: agreement and correctness can come apart. The authors frame this as a blind spot for label-free evaluation, with implications for anyone using such methods to rank models or select outputs without human labels.