A new paper on arXiv introduces a benchmark designed to evaluate cybersecurity agents that discover vulnerabilities, with a specific focus on business-logic flaws. The authors argue that synthetic benchmarks do not reflect real-world conditions, while real-life benchmarks are susceptible to a key failure mode—agents may succeed by memorizing known cases rather than by genuine discovery.
The benchmark's title, "Discovery or Memorization?," signals its central aim: to separate true vulnerability discovery from pattern recall. By grounding the evaluation in real-world business-logic scenarios, the work attempts to provide a more faithful test of agent capabilities than purely artificial setups.
As the abstract is truncated, full methodological details are not available, but the framing suggests a deliberate effort to address a recognized weakness in current evaluation practices. The paper is available on arXiv under identifier 2610.17542.