Language-model agents are increasingly evaluated on repository-level software engineering (SWE) benchmarks built from GitHub issues and pull requests, according to the abstract of a new arXiv paper. These benchmarks cover a broad range of development tasks, but the abstract cuts off before explaining how they fall short.
The paper introduces OS-Bench, a benchmark specifically for evaluating SWE agents on cross-OS code migration. The title suggests that moving code between operating systems is a distinct challenge not addressed by existing benchmarks.
Because the source only provides the abstract's opening lines, the full scope of OS-Bench—including its task design and evaluation metrics—is not described in the available text. The paper is listed as arXiv:2610.19368v1.