A new framework called EVA (Evaluating Voice Agents), developed by ServiceNow Research and Hugging Face, aims to standardize how voice assistants are tested. Unlike typical benchmarks that focus on single-turn speech recognition or basic question answering, EVA simulates realistic, multi-turn conversations. It includes 10 tasks that range from simple commands to complex dialogues, with added challenges like background noise and varied speaking styles.
The framework uses a large language model as an automated judge to score agent responses, and the authors report that these scores align closely with human preferences. This suggests EVA could serve as a reliable proxy for human evaluation, which is often expensive and slow. The benchmark is designed to be reproducible and extensible, allowing researchers to add new tasks or adapt it to different domains.
Initial results from EVA reveal that current voice agents handle straightforward requests well but falter on more intricate, multi-turn interactions. The authors argue that this gap highlights the need for better training and evaluation methods, pushing the field beyond simple command-following toward more natural, context-aware dialogue. The framework is open-sourced, inviting broader community participation in improving voice AI.