Reward models for reinforcement learning from human feedback (RLHF) typically rely on preference data that is far from perfect. The abstract of a new arXiv paper notes that noise enters from subjective human judgments, annotation mistakes, and imperfect LLM judges. When standard reward modeling is trained on such feedback, the resulting estimate of the clean risk is biased.

The paper proposes CleanRM, a reward modeling approach that combines noisy feedback with a small amount of clean feedback. Rather than treating all labels as equally trustworthy, the method uses the clean examples to anchor the training and correct for the bias introduced by the noisy majority. This aligns with the title's emphasis on needing only a few clean feedback points.

The authors argue that this setup is practical, since fully clean preference data is expensive to obtain but small curated sets are feasible. The work targets a core problem in RLHF: making reward models reliable when the feedback signal itself is unreliable. Further details of the method and experiments are not available in the abstract alone.