Researchers at Northwestern University ran AI coding agents against 112 real security bugs drawn from 84 open-source projects written in C/C++, Go, and Rust. They found that the common benchmark question—does the proof-of-concept input still crash the program?—misses the point. A patch can block the crash path without fixing the underlying flaw, a result the authors describe as worse than no fix because it creates false confidence that the problem is solved.
To separate real fixes from superficial ones, the team used the tests that project maintainers wrote alongside the actual fix. In one PHP example, a patch that threw errors on unusual inputs stopped the crash but broke legitimate calls, while another patch fixed the logic the way maintainers did; only the maintainer-written test distinguished them. With Claude Opus 4.7, the newest model tested, about a third of the patches that made the crash go away still failed those maintainer tests. Overall, it stopped crashes about 90% of the time but passed maintainer tests only about 61% of the time.
The evaluation setup also let agents find answers by accident or by design. One agent with web search enabled fetched the upstream fix commit within its first four tool calls; another reproduced an earlier fix from git history almost line for line; an earlier harness pointed an agent to the ground-truth patch via a configuration file. The authors note that searching git history is a reasonable move for a human developer, but the benchmark had left the answer in reach. A review of 13 benchmarks found that all five built for AI agents left at least one of nine pitfalls unaddressed. The article suggests asking vendors whether scoring runs maintainer tests, what access the agent had, what build and test instructions were given, and how timeouts and pipeline errors were counted.