Eugene Yan's survey of LLM-as-judge methods shows that large language models are increasingly used to evaluate other models' outputs, especially where traditional metrics fall short. The article covers common applications—such as summarization, instruction following, and conversational quality—and notes that these judges can scale and align with human preferences better than rule-based scoring in many cases.
Yan details several practical techniques to improve evaluator performance: prompting strategies like chain-of-thought reasoning, using multiple judges and aggregating scores, and finetuning judges on human-labeled examples to reduce systematic biases. He also stresses the importance of calibrating judges against human judgments before deployment, since agreement rates vary by task and model.
At the same time, the article does not ignore critiques. Known issues include positional bias (preferring the first or second response), self-preference when the judge is also the generator, and brittleness on creative or subjective tasks where clear rubrics are hard to define. Yan concludes that LLM evaluators are a powerful but imperfect tool—best used with explicit rubrics, multiple samples, and continuous validation against human raters.