User simulators are a common tool for training and testing conversational AI, but they often behave in ways that real users don't. Google Research's new framework, ConvApparel, tackles this problem head-on by first measuring how far a simulator's outputs diverge from actual human interactions, then using that measurement to guide improvements.
The approach works by comparing simulated user responses against a reference set of real user data across multiple dialogue dimensions. Instead of relying on a single overall score, ConvApparel pinpoints specific areas where the simulator is unrealistic—such as how it handles clarifications or expresses uncertainty—and then adjusts the simulator's policy to better match observed human patterns.
Because the source is a single blog post, there are no conflicting viewpoints to reconcile. The key claim is that this measurement-and-correction loop can be applied iteratively, making simulators progressively more faithful to real users. That matters for AI evaluation, since more realistic simulators mean more reliable predictions about how a deployed system will actually perform.