A new preprint on arXiv examines a growing practice in AI evaluation: using language-model user-proxies instead of human participants to measure how much people rely on an AI answer. The authors ask whether these simulated users can reproduce human reliance across contexts, and they are skeptical of the evidence so far.
The paper specifically targets the Rel-A.I. benchmark, arguing that published simulated-user studies do not establish validity for the kind of partial-answer decision involved. It also points out that Rel-A.I.'s released human record omits information needed to validate proxy behavior.
Because the abstract is truncated, the full methodology and results are not available from the source alone. What is clear is that the authors are raising a cautionary flag: before LLM proxies are treated as stand-ins for humans in reliance measurement, the field needs stronger validation evidence.