Reinforcement learning has become a standard way to sharpen the reasoning skills of large language models, but long-horizon training exposes a structural weakness. Because the model generates its own future inputs, small errors at an early step can accumulate, pushing the distribution of states far from what the model saw during training. This state distribution gap is a known challenge in RL, and it becomes especially acute for LLMs that produce extended autoregressive rollouts.
The new preprint, "Trajectory-Aware Policy Optimization: Closing the State Distribution Gap in Long-Horizon LLM RL," proposes a method that considers the trajectory as a whole rather than treating each step in isolation. The goal is to keep the policy's state distribution closer to the training distribution, mitigating the compounding effect of local mismatches. The abstract does not include experimental results, so the practical gains are not yet established.
As an arXiv announcement, this is an early release without peer review. The core idea — aligning training and rollout distributions over entire trajectories — addresses a real bottleneck in long-horizon LLM RL, but whether it holds up in practice will depend on the full evidence.