Speech large language models (SpeechLMs) can handle spoken requests, but they still underperform text-prompted LLMs on instruction following and reasoning. The authors of a new arXiv paper argue this gap is not just an ASR or representation problem: the same semantic instruction can trigger different response policies when spoken with different speakers, accents, noise, prosody, or disfluencies.

To address this, they propose Verifiable Invariant Reinforced Behavior Alignment (VIRBA), a reinforcement-learning framework that aligns SpeechLM behavior across acoustic realizations of the same semantic intent. VIRBA constructs multi-view spoken instruction groups, scores sampled responses using semantic preference, rule-verifiable correctness, cross-acoustic invariance, and adaptive reasoning rewards, and optimizes the model with Cross-Acoustic Group Relative Policy Optimization (CA-GRPO).

This moves alignment beyond teacher imitation toward robust reasoning policies that stay stable across realistic spoken variations. In experiments with recent SpeechLM baselines, VIRBA showed the largest improvements on reasoning-heavy and acoustically perturbed prompts, including disfluency robustness, spoken QA, audio reasoning, and speech-to-text translation. The paper is accepted at NeurIPS 2026.