A new preprint on arXiv proposes RM-OPD, an on-policy distillation method for training generative reward models (GRMs). The authors note that reinforcement learning on mixed-domain preference data (Mix-RL) has become the standard approach for building general-purpose GRMs, but they identify two critical bottlenecks in this pipeline. The abstract cuts off mid-sentence, so the exact nature of these bottlenecks is not fully described in the available text.
The proposed remedy, RM-OPD, appears to use on-policy distillation to improve the training process, though the abstract does not yet provide implementation details or experimental validation. Because the source is only a partial abstract, readers should consult the full paper for specifics on the method and its reported performance.
This is a single-source digest; no independent verification or comparison with other approaches is possible from the provided material.