Goldberger and Rudi tackle the problem of selecting the best k items from a large pool when evaluations are expensive. Their setting covers job applicants, LLMs, award nominees, and product ideas, where a decision-maker must budget human evaluations wisely. The paper allows both ratings and pairwise comparisons, arguing that the two forms work well together: ratings give a rough picture early, and comparisons then sharpen the ordering of the best candidates.
The authors' primary contribution is an adaptive allocation policy built on a forward-looking surrogate objective. Rather than only choosing evaluations that directly improve the current selection, the policy values evaluations that help decide what to evaluate in future rounds. The decision rule is interpretable, and the authors prove it is asymptotically consistent. Late in the process, when forward-looking choices are less useful, they switch to a myopic policy using a defined criterion.
In simulations, the method offers insight into how to combine ratings and comparisons and outperforms the limited existing benchmarks. The paper is available on arXiv as 2610.26380.