Large language models are increasingly used as judges, scoring open-ended outputs against checklist rubrics. These same rubrics also serve as reward signals during training. A new preprint on arXiv examines this practice, focusing on assumptions that are often taken for granted—such as how leniency or other biases shape the scores.
The paper proposes using Shapley attribution, a game-theoretic method, to decompose a judge's overall score into contributions from individual rubric items. This makes it possible to see which checklist criteria actually carry the standard, rather than assuming every item matters equally.
Because the abstract is truncated, the full findings are not yet visible. Still, the stated aim is clear: to move beyond treating checklist-based LLM judges as black boxes and toward a more transparent understanding of what drives their decisions.