Omni-modal large language models (OmniLLMs) are gaining attention for their ability to process audio and video together, but they face a practical bottleneck: long multimodal token sequences create substantial computational overhead. A new arXiv paper, OmniSelect, proposes a dynamic, modality-aware token compression strategy to address this issue.
Rather than applying a fixed compression rate, OmniSelect adapts compression based on the modality of the incoming tokens. This allows the model to retain important information across audio and video streams while reducing the number of tokens that need to be processed. The approach is designed to lower computational costs without sacrificing the unified understanding that OmniLLMs aim to achieve.
The paper is still a preprint, and the abstract does not provide full experimental details. However, the core idea—dynamic, modality-aware compression—offers a promising direction for making omni-modal models more practical for real-world use, where efficiency is often as critical as accuracy.