A paper accepted to IEEE SLT 2026 and posted on arXiv (2603.26246) asks whether multimodal context from earlier turns improves LLM-based automatic speech recognition. The authors find that, after supervised multi-turn training, conversational context mainly helps with contextual entities—terms or names introduced earlier in the conversation—rather than improving recognition uniformly.
Conditioning on raw audio context is expensive because the prior-turn audio embedding sequence grows rapidly as the conversation lengthens. To address this, the team proposes Abstract Compression, a learned latent abstraction that replaces each prior-turn audio sequence with a fixed number of latent tokens while retaining the corresponding transcript explicitly. This reduces the prior-turn audio footprint while keeping useful information available to the model.
On both in-domain and out-of-domain test sets, the compression-enabled model recovers part of the gains of raw-context conditioning. The paper also provides targeted analyses of the compression setup and its tradeoffs.