Policy gradient methods for autoregressive sequence models typically compute gradients by differentiating each token's log-probability while holding the conditioning history fixed. According to a new paper on arXiv, this standard approach throws away information about how earlier decisions shape the conditional distributions of later tokens.
The authors argue that this missing information matters for learning: when a model changes the probability of an early token, the downstream conditionals should also shift, but the conventional score function does not capture that effect. They propose reparameterizing the score function to make these dependencies explicit.
The paper is available as arXiv:2610.24188. As a single source, it presents one perspective; no conflicting findings are reported in the abstract.