Large language models are increasingly used for automatic program repair, but the benchmarks used to evaluate them may be unreliable. A key issue is data contamination, where test samples appear in training data, making results look better than they are.

To address this, researchers propose BenchChecker, a tool for assessing the credibility of bug-fixing benchmarks. It aims to detect and measure contamination, helping researchers trust evaluation outcomes.

The work highlights the need for rigorous benchmark validation in the LLM era. By flagging contaminated benchmarks, BenchChecker could lead to more honest assessments of model capabilities.