Standard LLM leaderboards often reduce performance to a single number, but a new arXiv paper argues that evaluation is inherently multi-objective. Different models may offer different tradeoffs across dimensions such as math, code, and writing, meaning a scalar ranking can hide the fact that one model is stronger in one area while weaker in another.
The paper proposes an adaptive sampling strategy designed to identify the Pareto frontier of model performance — the set of models where improving one capability necessarily sacrifices another. Rather than exhaustively comparing all models across all dimensions, the method aims to efficiently sample the most informative comparisons.
This is a preprint on arXiv, and the abstract does not include experimental details or validation. As the only source, the claims have not been peer-reviewed, so the approach's practical effectiveness remains to be seen.