Safety benchmarks for AI models usually boil down to a single number, but that number may obscure as much as it reveals. The new arXiv paper argues that because a benchmark suite is often composed of datasets targeting different safety-related attributes, two models with nearly identical overall scores can fail or succeed on very different things.
The authors describe this as a psychometric problem: benchmark scores should be treated as measurements of latent attributes, not as simple pass/fail totals. They conduct an audit of an existing safety benchmark to show how aggregate reporting can hide attribute-level weaknesses, and they argue for reporting and evaluating safety performance in a more granular way.
The paper does not claim that current benchmarks are useless, but it does suggest that relying on headline scores alone can give a misleading picture of model safety. The implication is that researchers and policymakers should look beyond the aggregate and ask which specific safety attributes a score actually reflects.