TokenGate, described in a new arXiv preprint, takes inspiration from expert human readers, who skim long texts and focus only on what matters. Standard language models, by contrast, process every input token through every layer during prefill, which becomes costly as inputs grow.
The paper proposes learned token pruning: the model learns which tokens are important and prunes the rest before they are processed further. The authors position this as a way to make prefill faster and more memory-efficient for large inputs such as code repositories and long documents.
Because the source is a single arXiv abstract, the article does not compare competing approaches or experimental results; it simply reports the proposed method and its stated motivation.