The policy gradient theorem gives the gradient of expected discounted return with respect to policy parameters. Yet many practical policy gradient methods drop the outer discount factor, producing an update that is not the gradient of the discounted objective. Whether this modified update was the gradient of some other objective had remained an open question for years.

Chris Nota and Philip Thomas now answer that question for the episode-summed, on-policy update: it need not be the gradient of any function. They construct a counterexample where exact expected updates push the policy toward a deterministic limit that is globally pessimal for both discounted and undiscounted returns, while the parameter itself diverges.

The authors also survey the literature and find widespread misunderstanding about discounted policy gradient methods, including errors in highly cited papers at top conferences. Their result clarifies a subtle but consequential gap between theory and common practice in reinforcement learning.