A new arXiv preprint takes up the question of whether AI helps cyber attackers or defenders, but with a twist: it argues that the standard way of measuring AI cyber capability may be flawed. Public vulnerability benchmarks, the authors note, can leak the very information an AI agent needs—published advisories, exploits, and fixes—making it hard to tell whether the AI is genuinely skilled or just retrieving known answers.
To get around this, the study turns to nonpublic vulnerabilities and looks at whether subsequent attacks follow. That shift, the abstract suggests, offers a more realistic test of how AI systems behave in the wild, away from the curated environment of public benchmarks.
The stakes are practical. Frontier AI release decisions increasingly hinge on cyber capability benchmarks, and if those benchmarks are contaminated, they could give a false sense of either risk or safety. The paper does not settle the attacker-defender debate, but it does highlight a methodological blind spot worth fixing before those benchmarks are used to gate deployment.