A new preprint from arXiv argues that language priors in multimodal large language models (MLLMs) can override visual evidence, leading to impaired visual perception. The authors propose a new benchmark and mitigation strategy, but the core finding is that models often answer based on what they expect to see rather than what is actually in the image.
The paper criticizes existing benchmarks for assuming that all models share the same priors and for measuring performance only through accuracy. According to the abstract, cross-model prior agreement falls below 80%, meaning that different models rely on different priors in many cases. This undermines the validity of accuracy-only evaluations, since a model might score well by guessing according to its own priors rather than by truly perceiving visual input.
The abstract is truncated, so details on the proposed mitigation are not fully available. Still, the authors clearly aim to address the interference of language priors, suggesting that future MLLM evaluation should account for per-model priors and go beyond simple accuracy metrics. This is a single-source report, so no conflicting findings are noted.