A position paper posted on arXiv argues that standardized item-level benchmark data should become the default infrastructure for AI evaluation. The authors contend that current evaluation practices are hampered by incomplete transparency about the specific items used in benchmarks, making it difficult to reproduce results or understand what models are actually being tested.
The paper identifies several recurring problems, including underspecified item selection and construct misalignment—where the benchmark items do not clearly correspond to the capability being measured. These issues, the authors say, weaken the conclusions drawn from AI evaluations and limit the ability of researchers to compare systems fairly.
As a remedy, the paper calls for routine release of item-level data in a standardized format, so that every benchmark includes the full set of items, their properties, and scoring rules. The authors argue this would improve reproducibility, allow for more granular analysis, and help align evaluation design with intended constructs. Because the abstract is truncated, the full set of proposed standards is not visible, but the core argument is clear: without item-level transparency, AI evaluation remains underspecified.