Hybrid architectures have become a popular way to build efficient language models, pairing a small number of full-attention layers with a larger number of cheaper sequence layers like sparse attention and recurrent layers. This design cuts computational cost, but it often comes with a trade-off in expressive power. A new paper on arXiv, titled "Improving the Expressive Power of Hybrid Sequence Models via Multi-Hop Retrieval," tackles that limitation directly.
The authors propose adding a multi-hop retrieval mechanism to the hybrid framework. Instead of relying only on the local or sparse context that efficient layers typically see, the model can retrieve relevant information from earlier in the sequence and pass it forward through multiple steps. This lets the model reason over distant dependencies more effectively, potentially giving it some of the flexibility of full attention without the quadratic cost.
The abstract does not include experimental results or implementation details, so the claims are not yet verified. Since this is the only source, there is no alternative view to compare against. Still, the idea offers a plausible route for making hybrid models more capable while preserving their efficiency advantage.