Most vision-language models encode every image with the same fixed token budget, regardless of what question is being asked. A new paper from researchers at the University of Waterloo argues this is wasteful: the visual evidence needed to answer "what color is the car?" is not the same as for "how many people are in the background?" They propose QART, a query-adaptive recurrent tokenizer that steers representation toward query-relevant regions and grows the token count only until those features are faithfully reconstructed.
QART starts from frozen encoder features and recurrently builds a compact representation. A reconstruction loss over the query-relevant visual support controls when to stop adding tokens. This makes the module plug into existing backbones without retraining the encoder. The authors report 81.96% accuracy on Visual7W and 76.37% on VQAv2 using only about 29 visual tokens, cutting token counts by 20–25x relative to dense VLM inputs. On retrieval benchmarks, it reaches 92.0% P@1 on CORE and 36.3% PR-AUC on PODS, beating SteerViT by 8.4% on the latter.
The results suggest that visual representations need not be fixed in content or capacity. By tying both to the query, QART offers a practical way to reduce compute in vision-language systems without sacrificing accuracy. The paper is a single-source preprint, so independent replication and comparisons on broader benchmarks are still needed, but the core idea—budget-adaptive, query-steerable tokenization—is a clear departure from the fixed-token paradigm.