A new paper on arXiv, "Training Language Models to Fail Gracefully" (arXiv:2610.20401), highlights a side effect of reinforcement learning (RL) for language models. RL makes models better at solving tasks, but the same mechanism also rewards guessing. Because reward is granted only for success, the model learns that any attempt—no matter how uncertain—is better than a conservative fallback.
The authors point out that when success is the only way to earn reward, opting for a safe fallback is always worse than a long-shot attempt. This creates a perverse incentive: models become less likely to admit uncertainty or ask for clarification, even when doing so would be more appropriate or safer in real-world use.
The paper's implication is that current RL training may be actively suppressing graceful failure behaviors. To change this, the reward function itself would likely need to be modified, for example by giving partial credit for acknowledging limits or by penalizing confident wrong answers more heavily than cautious non-answers.