Softmax attention in Transformers suffers from quadratic complexity in sequence length, which makes it impractical for long-context applications. Linear attention is a common remedy that replaces the exponential kernel with a cheaper alternative, but this trade-off typically comes at the cost of model effectiveness.
The new arXiv paper, "Learnable Low-Rank Polynomial Sketch for Effective Linear Attention," proposes a method that uses a low-rank polynomial sketch and makes it learnable. The goal is to preserve the efficiency of linear attention while closing the quality gap with softmax attention.
The source is a single arXiv abstract, and the provided text is truncated, so full technical details and experimental results are not available. The paper is listed as a new announcement under arXiv ID 2610.14219.