Software engineering benchmarks typically place an agent in a fixed code repository and judge it on hidden tests. The new paper argues that this setup ignores a key variable: the starting state of the environment. If agents are only evaluated from one initial configuration, their measured performance may not reflect real-world conditions.

The authors propose SWE-Residual, an evaluation method that systematically varies the starting state to test whether coding agents can still complete tasks. This shifts the focus from raw performance on a single setup to robustness across plausible starting points. The work highlights a gap in current evaluation practices, suggesting that coding agents need to be tested under a range of initial conditions to better understand their practical reliability.