A new preprint from arXiv highlights a subtle flaw in how LLM review pipelines are evaluated. The authors tested a setup where Qwen3-8B reviews answers produced by Llama-3.1-8B on math problems. The review process appeared highly effective, boosting the generator's accuracy by 33.2 points. That gain looks impressive at first glance.

However, the researchers point out a critical confound: Qwen3-8B on its own, without any reviewing step, was 6.4 points more accurate than the final reviewed output. In other words, the reviewer's apparent contribution is partly just its own superior capability leaking through, not a distinct verification skill. The common practice of measuring a reviewer by its gain over the generator therefore gives a misleading picture.

The paper argues that verification and capability transfer should be separated when designing and benchmarking LLM pipelines. Without that separation, a strong model used as a reviewer may look like a great verifier simply because it is a great solver. The authors suggest that future evaluations need to account for this baseline to avoid overestimating the value of review mechanisms.