Standard transformer attention scales quadratically with sequence length, a bottleneck for long inputs. A new arXiv preprint (2610.20637) tackles this by expanding the exponential kernel of attention into a truncated Taylor series of n terms. Each term is then factorized into a tensor-power inner product, which lets the model compress the entire causal history into exactly n recurrent states that update as new tokens arrive.

The authors show that this construction yields linear-time complexity while maintaining a fixed number of states, independent of sequence length. This differs from some prior linear-attention methods that require growing memory or approximations with less clear error bounds. The Taylor-series route provides a principled way to trade a small amount of kernel fidelity for a large gain in efficiency.

Because this is a single preprint, there is no independent verification or contrasting source to compare against. The claims are theoretical and algorithmic, and the abstract does not report experimental results. Still, the approach offers a concrete path toward transformers that can process very long sequences without the usual quadratic blowup.