Vision-language models are typically compared on leaderboards that collapse performance into a single ranking. A new arXiv preprint, Shift-Reliance Alignment: Ranking Vision-Language Models by the Cues They Trust, argues that this one-ranking-fits-all approach can be misleading for at least one important class of questions.
The paper highlights questions about objects that a scene suggests but does not contain. In these cases, the model with the lowest overall error on a leaderboard may not be the one that handles such ambiguous visual cues best. The available abstract is truncated before presenting results, so the full method and findings are not yet described in the source.
The proposed direction is to rank models by the cues they trust, rather than by aggregate accuracy alone. That would mean aligning a model's ranking with the specific visual evidence it relies on, which could give a more useful picture of when a model is likely to fail.