Conversational search systems are difficult to evaluate at scale. Human studies are limited, and user simulators offer a way to run larger evaluations—but only if the simulated users behave realistically. A new arXiv paper, announced as AutoSim, takes up this problem.
AutoSim is described as a self-improving user simulator for conversational search. The abstract stresses that the value of such a simulator depends on whether its behavior is realistic enough to trust. Without that trust, large-scale evaluations would be hard to rely on.
The available text is an abstract, so the paper's technical details are not yet clear from the announcement. The snippet does not include comparisons to other simulators; it simply frames the need for realistic behavior and introduces the system as the subject of study.