Human conversation is not just words; it relies on gestures and facial expressions that accompany both speaking and listening. Most dialogue systems focus on spoken responses, ignoring this multimodal layer. A new arXiv paper introduces ELLSA 2, an end-to-end full-duplex dialogue model designed to coordinate speech, gestures, and expressions in real time.

ELLSA 2 uses an autoregressive, chunk-wise synthesizer that combines pretrained acoustic features with separate hidden states for listening and speaking. This lets the model produce continuous non-verbal behavior even during silence. For semantic gestures, the authors introduce contextual semantic calls and template guidance, which separate the decision to use a gesture from how to execute it.

The authors built a curated data and evaluation pipeline for dialog-aware co-speech gesture generation and used reference-based multimodal LLMs as automatic judges. Experiments show ELLSA 2 remains competitive with open-source full-duplex speech models and outperforms cascaded baselines on gesture and expression metrics. The reference-based scores also showed strong system-level correlation with human judgments, suggesting they could serve as useful preliminary metrics for full-duplex and semantic gesture assessment.