TRACE, a new benchmark from Morais and colleagues, targets a blind spot in how large audio-language models (LALMs) are evaluated. Most benchmarks ask only whether the final answer is correct, which can mask whether a model relied on acoustic evidence or on language and contextual priors. TRACE instead measures three dimensions: whether the answer is right, whether the model attends to the audio regions that actually matter, and whether answers shift coherently when the audio is altered in controlled ways.
The benchmark focuses on temporal ordering, counting, and event existence, using synthesized scenes that are contextually plausible but contain known ground truths. This design makes it possible to test whether models hallucinate events implied by the context but absent from the audio, and to generate programmatic variants that stress-test robustness. Attribution-based metrics quantify how much attention models pay to target events versus non-target events across many scenes.
Results from five state-of-the-art models show a consistent pattern: they often detect where relevant sound evidence occurs, but they do not reliably interpret that evidence, integrate it into answers, or respond appropriately when the evidence changes. Ordering and counting questions proved especially difficult. The authors argue that correctness alone is insufficient for trust, and that evidence alignment and robustness should become standard diagnostic dimensions for audio-language systems.