Evaluating software agents on a full benchmark is costly, which pushes researchers to use small task subsets. According to a new arXiv paper, these subsets can preserve the relative ranking of agents, but they yield scores that are not accurate in absolute terms. The paper introduces HSPR (Hierarchical Posterior Inference) as a way to address this problem.
The method exploits historical responses from previous evaluations, which provide useful structure for inference. The abstract notes that tasks with identical source responses are a key part of this structure, though the full details are cut off in the available text. HSPR appears to combine the ranking information from the subset with historical data to produce more reliable score estimates.
This work is relevant to anyone building or benchmarking software agents, as it offers a potential path to cheaper evaluation without sacrificing score accuracy. The paper is available on arXiv under identifier 2610.24611.