Visual token pruning is a common technique for reducing the inference cost of vision-language models (VLMs) by discarding some visual tokens before they are processed. The goal is to keep the model's capabilities intact while making it cheaper to run. But according to a new preprint on arXiv, the field may be measuring success in the wrong way.
The paper, Are We Evaluating VLM Token Pruning Right? Looking Beyond Accuracy and Token Count, argues that existing evaluations rely on aggregate task accuracy and retained token count as proxies for pruning quality. Those summary numbers, the authors contend, do not capture the full picture of how a pruned model actually behaves. The abstract specifically calls for looking beyond accuracy and token count, suggesting that current benchmarks may reward methods that look good on aggregate statistics but are less informative about real-world trade-offs.
Because this digest draws on a single preprint, there is no independent result to compare; the claims reflect the authors' own framing. Still, the critique points to a useful question for the field: whether pruning research is optimizing the right evaluation targets, not just the headline numbers.