Recent advances in large vision-language models have produced strong results on visual question answering and multimodal reasoning. But the authors of a new paper argue that it is still unclear whether these models are genuinely performing grounded visual reasoning, or merely exploiting surface patterns.

To address this, they introduce MagiC, a new evaluation method designed to assess multimodal cognition with a focus on grounded visual reasoning. The paper, posted on arXiv, presents MagiC as a tool for probing whether models truly understand the visual grounding behind their answers.

The abstract does not provide details on the benchmark's tasks or results, but the motivation is clear: separating genuine visual understanding from impressive but shallow performance.