When a Reddit user asks for recommendations, replies typically name different titles. For an LLM recommender trained with reinforcement learning, that creates a problem: which reply should the model be rewarded for matching? The new arXiv paper frames this as responder noise, where one training example has many right answers rather than a single target.

The paper focuses on GRPO, a reinforcement learning algorithm that evaluates a group of sampled responses. The authors argue that the training signal should not be tied to any one human reply, and instead should account for the variety of valid answers present in the data. Removing that noise, they suggest, should give the recommender a more accurate learning signal.

Because the abstract does not include experimental results, the contribution here is primarily in how the training problem is posed and addressed. Still, the issue is not limited to Reddit threads: any RL setup where human demonstrators disagree could benefit from treating the disagreement as noise rather than as a single ground truth.