LLM-as-a-judge is increasingly used to evaluate model outputs, but its reliability is threatened by self-preference—the tendency of a judge to favor its own generations. A new arXiv paper (2608.18091) investigates this issue further, focusing on how labels of "self" and "other" affect judge behavior.

The abstract notes that self-preference has been studied predominantly in one direction. The paper's title and abstract suggest the authors find a broader, bidirectional bias: judges are influenced not only by recognizing their own outputs but also by labels identifying other models' outputs. Specific experimental details are not available in the source, as the abstract is truncated.

This matters for evaluation design. If judges are biased by knowing the provenance of outputs, benchmarks relying on LLM judges may need to blind or otherwise control for label information. The work is an arXiv preprint, so findings should be considered preliminary until peer review.