Multimodal benchmarks are widely used to test whether AI models can reason across text, images, and other modalities. But a new preprint argues that some of these benchmarks may be less reliable than assumed, because they contain shortcut items that models can answer correctly without actually integrating information from multiple modalities.
The paper, posted on arXiv, proposes a diagnostic framework called Structure-Response Co-Calibration. This method is designed to examine benchmarks at the individual item level, rather than treating a benchmark as a single score. By calibrating the structure of each item against the responses it elicits, the approach aims to flag cases where a correct answer does not reflect genuine multimodal reasoning.
The authors note that existing literature has already pointed to the presence of such shortcuts. Their contribution is a systematic way to identify and characterise these problematic items. If adopted, the method could help benchmark designers remove or redesign shortcuts, leading to evaluations that more faithfully measure a model's multimodal capabilities.
As with any preprint, the findings have not yet been peer-reviewed, and the full details of the co-calibration procedure are not available in the abstract. Still, the work highlights a growing concern in the field: that high benchmark scores may sometimes overstate a model's true understanding of multimodal data.```json {