Large audio-language models (LALMs) have grown better at connecting sound to language, but their ability to do low-level listening—catching details like pitch and duration—remains largely untested. A new benchmark, WoW-Bench, fills that gap by using marine mammal vocalizations as unfamiliar sound events. Because these sounds are outside the models' typical training distribution, the benchmark forces models to rely on acoustic cues rather than prior knowledge of the sound source.
The benchmark is split into two parts. The Perception benchmark asks models to categorize novel sounds, while the Cognition benchmark, inspired by Bloom's taxonomy, probes remembering, understanding, applying, and analyzing sound events. To make the test harder to game, the authors add distractor questions that check whether a model is genuinely solving the task through listening or instead using shortcuts such as textual heuristics.
Experiments with current state-of-the-art LALMs show performance well below human levels, with the gap widening on cognition tasks. The authors argue this indicates a need for stronger auditory grounding in LALMs, and they hope WoW-Bench will help drive progress on fine-grained acoustic perception for real-world, out-of-distribution audio understanding.