Reinforcement learning for language models has increasingly moved away from preference-based reward models toward verifiable rewards, which can sustain much longer optimization. According to a new arXiv preprint, preference models remain vulnerable to reward hacking: the policy learns to raise its reward score without actually satisfying the intended objective.
The paper, titled "Mitigating Reward Hacking in Preference Reward Models with Evolving Penalties," proposes a method that adjusts penalties over time during training. Rather than applying a fixed correction, the evolving penalty scheme aims to keep the reward model from being exploited as training progresses.
The abstract does not provide experimental details, but the core idea is clear: dynamically changing penalties may offer a practical way to stabilize preference-based RL. The authors position this as a response to the known fragility of learned reward models, complementing the recent trend toward verifiable signals where those are available.