Failure-attribution benchmarks are designed to test whether an LLM judge can pinpoint which step or agent in a pipeline caused a failure. The intended meaning of a high score is that the judge did more than guess: it checked the faulty step against the run itself, a cross-step verification.

A new auditing paper on arXiv argues that this interpretation may be too optimistic. The authors point to spurious signals—surface cues in the data—that could let a judge achieve high scores without actually performing the cross-step check. If those cues are predictable from the label, benchmark scores may reflect pattern matching rather than genuine causal diagnosis.

The abstract is truncated, so the full methodology and experimental results are not yet visible. But the core warning is clear: before trusting a failure-attribution score, we need to audit whether the judge is really checking the run or just latching onto surface regularities.