Reinforcement learning has become a common post-training technique for improving the accuracy of large language models on verifiable tasks. However, as the abstract notes, such training can narrow the set of solutions the model is able to produce, reducing its behavioral variety even as performance improves.
The paper introduces EliteReplay, a method that incorporates quality-diversity search into RL post-training. Quality-diversity search maintains a collection of solutions that are both high-performing and behaviorally distinct, rather than converging on a single optimal strategy. This is intended to preserve a broader range of successful outputs during fine-tuning.
The abstract mentions that several approaches have attempted to address this diversity problem, though the available text is truncated. EliteReplay appears to be one such attempt, specifically targeting the preservation of diverse solutions while retaining the accuracy gains from reinforcement learning.