Rubric-based evaluation is a common way to assess large language models: responses are checked against explicit criteria, and the results are aggregated into a score. A new paper on arXiv argues that this approach has a blind spot. Not all criteria carry the same weight, yet existing methods usually treat them as if they do.
The authors propose an importance-aware criteria weighting scheme, which adjusts the contribution of each criterion based on its relevance to the overall quality being measured. The abstract does not detail the exact mechanism, but the motivation is clear: a rubric that weights all items equally may misrepresent how well a response actually performs.
If the method holds up, it could make LLM evaluation more faithful to human judgment, where some aspects of a response matter more than others. The paper is a single source, so no independent comparison is available, but the direction is notable for anyone building or using rubric-based benchmarks.