Benchmarks are central to measuring AI progress, but poorly designed tasks can distort results. A new arXiv paper, “Between Spoilers and Guesswork: Avoiding Over-Specification in Coding Benchmarks,” identifies two opposing failure modes: task under-specification and task over-specification.

The authors argue that under-specification leads to guesswork, where models invent missing details or, in worse cases, rely on data contamination or reward hacking. Over-specification, on the other hand, can leak information that properly designed tasks should require the model to deduce.

To make benchmarks more reliable, the paper advocates for a middle ground—giving enough specification to constrain solutions without turning tasks into spoilers. The claims come from a single source, so no independent comparison is available, but the framing highlights a subtle trade-off in evaluation design.