A new arXiv preprint, “From Software History to Executable Regression Tasks: A System for Evaluating Software Agents,” tackles the challenge of assessing AI agents that write and change code. The authors argue that as these agents become more common, it is crucial to test their ability to review changes and find defects, not just generate code.

The proposed system converts software history into executable regression tasks. This gives agents a concrete project environment to investigate, rather than relying on static benchmarks. The goal is to create a reliable way to measure whether an agent can catch regressions or bugs introduced during development.

The abstract is brief and does not specify the implementation, dataset, or experimental results. Still, the core idea is clear: use real project history to build dynamic evaluation scenarios. This could complement existing static code benchmarks by testing agents in more realistic, execution-based settings.