Reinforcement learning with verifiable rewards typically works by having a model try a task multiple times and then optimizing toward the successful attempts. That breaks down when tasks are so hard that the model almost never succeeds, leaving no successful rollouts to learn from. In a new arXiv preprint, researchers introduce RLTLDR, a method that turns each failed attempt into a learning signal by asking the model to write a single TL;DR insight based on the verifier's output.

The key idea is to condition the next rollout on all previously written insights, then backpropagate through those in-context insights so the model learns a direct mapping from task to insight. On challenging tool-calling and coding datasets filtered to have Pass@128=0, standard GRPO training of a Qwen 3.5 9B Thinking policy stays flat at 0–1% Pass@1. RLTLDR breaks through, achieving 14–31% Pass@1 when insights are present in context during training, and crucially 12–13% even when no insight is provided at evaluation time.

The authors also test a stripped-down version, SFTLDR, which trains only on (task, insight) tuples without showing any rollouts. Training on just 4,000 such tuples recovers almost the full performance of RLTLDR and of classical supervised fine-tuning on full rollouts. This points to a compact training paradigm of the form "on this sort of task, keep this sort of thing in mind," which the authors hope will inspire further research.