Evaluation benchmarks are meant to measure model capabilities, but their scores can hide a deeper question: who decided what counts as good performance? According to an abstract posted on arXiv, a new "user-centric audit framework" directly addresses this problem. Instead of treating benchmarks as neutral yardsticks, the framework examines the conventions, contexts, and judgments built into them.
The abstract states that benchmark scores can "obscure whose conventions, contexts, and judgments define successful performance." The proposed audit is designed to expose those hidden assumptions by centering the user in the evaluation process. The abstract does not detail the mechanics of the audit, so the contribution at this stage is primarily the framing itself.
The significance, as presented, is that benchmark results are not just technical outputs; they encode choices about which values matter. By making those choices visible, the framework could help researchers interpret scores more critically and design evaluations that better reflect the people they are meant to serve.