Large vision-language models (LVLMs) repeatedly process hundreds of visual tokens across decoder layers, which adds up to substantial computation and memory costs. The arXiv abstract for CatchUp highlights this repeated processing as a central inefficiency in current models.
A common remedy is inference-time visual token pruning, which removes tokens to reduce the workload. However, the abstract notes that existing pruning methods are often irreversible: once a token is pruned, it is gone for good, potentially losing information the model could have used.
CatchUp takes a different approach by introducing depth-aware re-admission of visual tokens. Rather than permanently discarding pruned tokens, the method allows them to be re-admitted at later layers, depending on the depth. This is intended to preserve useful visual information while still lowering the overall computational burden.