Vision-language models (VLMs) process images by converting them into tokens, but not all image tokens are treated equally. A new preprint on arXiv studies "visual attention sinks" — image tokens that consistently absorb a large share of attention no matter what the model is asked to do. The authors investigate how to detect these sinks and how to mitigate their effects.
The paper's title, "Attention Sinks in VLMs Are Borrowed, Not Seen," suggests that these attention hotspots are not learned from visual features. Instead, they appear to be inherited from the language model's pretraining, where certain positions or tokens act as default attention targets. If that interpretation holds, visual attention sinks are artifacts of the architecture rather than meaningful signals about the image content.
The authors propose methods to identify such sinks and to reduce their impact. The abstract is brief, but the framing points to a practical concern: if VLMs are paying attention to the wrong parts of an image, their visual reasoning may be less grounded than it appears.