This paper examines a common trick in reinforcement learning: dividing a trajectory's reward by its length to normalize for episode duration. The authors show that this normalization is not a neutral scaling—it can flip the direction of a reward-centered policy update, turning an ascent into a descent.
The analysis works in a finite trajectory model with a three-outcome softmax policy. Within that setting, the authors derive an exact threshold that determines when a length-normalized single-rollout update is guaranteed to ascend the expected reward. The threshold gives a precise condition for when the normalization is safe and when it is not.
The result is theoretical and limited to the specific model, but it highlights a subtle interaction between normalization and policy improvement. Practitioners using length-normalized returns should be aware that the update direction is not invariant to this choice.