A new arXiv paper highlights a growing problem in AI evaluation: benchmarks are saturating. As models rapidly improve, several of them now approach perfect scores on established tests, and the paper warns that this erodes a benchmark's core purpose.

Once a benchmark can no longer separate models by performance, it stops being a useful measurement tool. The abstract specifically notes that this loss of differentiating power also undercuts the benchmark's value for AI safety cases, which rely on reliable comparisons to assess risk.

The paper's framing suggests that evaluation methods need to keep pace with model capabilities. But the abstract does not propose specific alternatives, so the main takeaway is diagnostic: current benchmarks are becoming less informative as AI advances.