Large language model (LLM) evaluators are increasingly being used to assess personalized assistants, but whether they can reliably judge personalization has rarely been tested. A new paper introduces PersEvalBench, a benchmark designed to address this gap.

The benchmark specifically tests whether LLM evaluators can judge contextually appropriate use of persona information. This is a critical capability if such evaluators are to be trusted in evaluating assistants that tailor responses to individual users.

By providing a standardized test, PersEvalBench aims to measure how well LLM evaluators perform on this task, offering a way to assess their reliability in the personalization domain.