Reinforcement learning with verifiable rewards (RLVR) has been credited with improving model performance beyond the specific domain its rewards check. For example, training on mathematics has been reported to boost accuracy on MMLU, a general knowledge benchmark. But a new arXiv preprint, "Same Answer, New Box: How Much of RLVR's Out-of-Domain Gain Is Answer Format?", asks whether those gains are as substantive as they appear.
The paper's central question is whether the measured improvements come from better reasoning or simply from a change in how the model formats its answers. The abstract notes that accuracies are "usually read out of the model's text," which raises the possibility that RLVR may teach models to present answers in a way that is easier to parse or grade, rather than to reason more accurately.
If that is the case, some of the apparent out-of-domain benefits of RLVR could be an artifact of evaluation rather than a genuine transfer of skill. The preprint highlights the need for careful controls when interpreting RLVR results, though the full findings are not yet available from the abstract alone.