Speech large language models (LLMs) typically use pretrained acoustic encoders to convert audio into a sequence of representations that a decoder processes. However, LLMs scale poorly with input sequence length, and acoustic encoders tend to produce lengthy outputs—creating a computational bottleneck for tasks involving long audio.
The paper introduces ASSET, a method that frames speech segmentation as a reinforcement learning problem. Instead of using fixed-length chunks, ASSET learns where to split the audio adaptively, aiming to reduce the number of tokens fed to the LLM while preserving the information needed for downstream tasks.
By making segmentation adaptive, ASSET could lower the cost of running speech LLMs and enable them to handle longer inputs more efficiently. The abstract presents this as a way to improve scalability, though specific experimental results are not detailed in the available text.