Web-GUI agents that rely on vision-language models typically process high-resolution screenshots and long multi-turn histories, generating a flood of visual tokens. That makes inference slow and expensive. Most token pruning methods were designed for natural images and score tokens independently, which can break up the local visual structure needed to recognize small interface elements like buttons, icons, and hyperlinks.
STP-GUI, a framework introduced in a recent arXiv preprint, instead performs spatially coherent window-level selection. A lightweight selector is trained with policy learning while the VLM backbone stays frozen, allowing the model to keep fine-grained, instruction-conditioned token scoring even under aggressive reduction.
On the Online-Mind2Web benchmark, the method retains only 21% of tokens yet preserves 77.7% of full-token performance, while reducing inference latency by 31% and FLOPs by 57%. Tests on Android in the Wild also suggest the approach transfers to mobile GUI environments, pointing toward broader applicability beyond desktop web interfaces.