Large language models are increasingly used as automated judges in AI evaluation pipelines, yet one key setting — decoding temperature — is often selected by habit or guesswork rather than evidence. A new paper on arXiv takes aim at this gap, arguing that temperature deserves deliberate, data-driven attention.
The authors report running more than one million evaluations across multiple open-source models. That scale matters: it turns a hyperparameter that is usually treated as a footnote into a variable worth studying systematically.
Because this digest draws on a single source, there are no conflicting findings to compare. The paper's central claim is straightforward: setting temperature is necessary for reliable LLM-as-a-judge results, and leaving it to intuition is a risk.