Context extension for language models often relies on training with natural long documents, but the choice of which documents to use has received little systematic scrutiny. The new arXiv paper reports a controlled comparison of data-selection pipelines, aiming to identify what actually helps when extending a model's context window.
The study is built around Qwen2.5-3B, targeting a 128K context length, and holds the continued-training budget fixed at 6B tokens across all pipelines. That matched-budget design is meant to isolate the effect of document selection from the effect of training compute, making differences in outcome attributable to the data pipeline itself.
The provided abstract does not detail results, so this article can only describe the study's design and motivation. If the full paper delivers on its premise, it could give practitioners a clearer basis for choosing long-document data when extending context.