Curriculum learning usually arranges training samples from easy to hard, but the optimal difficulty schedule for reinforcement learning methods like GRPO (Group Relative Policy Optimization) is not obvious. A new arXiv paper, titled "A Step Harder, Not a Step Too Far," investigates how difficulty progression affects GRPO training.
According to the abstract, medium-difficulty samples are particularly attractive for online selection in GRPO because they are more likely to produce informative group-relative learning signals. This suggests that the most useful samples are neither too easy nor too hard, but sit in a middle range where they can still guide policy updates effectively.
The paper's title indicates that the authors identify both benefits and limits to curriculum difficulty progression: incremental steps may help, but jumping too far in difficulty can be counterproductive. The research adds nuance to how curriculum design should be approached for GRPO and similar methods, though the abstract provides only a brief glimpse of the full findings.