A new arXiv paper takes aim at a common practice in natural language processing: using correlation with human ratings to decide whether an automated metric or LLM judge is trustworthy. The authors argue that this approach, known as meta-evaluation, is built on a shaky foundation.
The paper focuses on multilingual style-personalized generation, where models are asked to produce text that matches a particular style or persona. In such settings, the choice of evaluation metric is especially consequential, yet the standard correlation-based checks may not reveal which metric actually works.
The abstract cites prior work showing that correlation is not an appropriate estimate of a metric's evaluation capability. It does not, in the abstract, describe the alternative the authors propose, but the title—'Fantastic Metrics and How to Find Them'—suggests a search for better meta-evaluation methods.