Long-context language models now handle ever larger inputs, and benchmarks often use context length as the primary measure of task difficulty. A new arXiv preprint, "Beyond Context Length: Characterising the Difficulty of Long-Context Reasoning," challenges this assumption.
The paper argues that context length alone cannot explain why models fail on long-context reasoning. The authors propose that difficulty is more complex, but the abstract does not specify which additional factors they consider.
The central claim is clear: evaluations should move beyond raw context length to capture what actually makes long-context reasoning hard.