Primal-dual algorithms are common for enforcing constraints in reinforcement learning, but their response to slowly changing environments is not fully mapped. Maxime Baelde's preprint on arXiv analyzes a two-timescale primal-dual method for constrained Markov decision processes, considering non-stationarity that enters either additively, through an oscillating constraint level, or parametrically, through an oscillating constraint sensitivity. The analysis is carried out on a tangent linear-quadratic reduction around an active saddle, so the results are local.
On the additive channel, the dual variable's oscillation saturates at a floor set by the moving optimum and is amplified linearly by the timescale ratio. On the parametric channel, a coherent modulation alone does not create resonant forcing: the stiffness modulation from the dual update cancels the damping modulation from the primal update. Instead, resonance appears when the constraint estimate the primal update uses differs from the measurement the dual update uses, producing a parametric resonance at twice the natural frequency. The tolerable mismatch grows with the square root of the forcing bandwidth and collapses with the square root of the dual-to-primal step-size ratio.
If both updates are driven from the same estimate, the resonant forcing disappears but a stationary offset equal to the estimator's bias remains. In a Lagrangian soft actor-critic agent, the multiplier locks onto the sub-harmonic at the timescale ratios predicted by its identified linear loop, while a coherent cost critic suppresses that lock-in. A proportional-integral-derivative term was predicted to leave the lock-in in place, yet suppresses most of it through projection of the multiplier. The findings highlight that the interaction between estimator design and step-size ratios, rather than modulation depth alone, governs stability in this setting.