Cybersecurity benchmarks for AI agents have largely rewarded offensive behavior, according to a new arXiv paper. The authors of CWE-bench argue that demonstrating an exploit is not the same as making software safe.
The paper highlights a fundamental asymmetry: an attacker only needs to find one vulnerability, while a defender must identify and fix every one. This makes defense a much harder test of an agent's capability.
To address this, the paper introduces CWE-bench, a benchmark designed to evaluate agents on defensive security tasks. By refocusing on repair and mitigation, it aims to measure whether agents can actually improve software security, not just break it.