AI agents are becoming increasingly capable in both offensive and defensive cybersecurity roles, but the benchmarks used to measure them have lagged behind. Most current evaluations place an agent in isolation and test it against a fixed, unchanging target. That setup does not capture the interactive, adversarial conditions of actual cyber operations.
CyberArena, a new benchmark described on arXiv, takes a different approach. It stages live, symmetric combat in which AI agents must simultaneously attack and defend. This design forces agents to adapt to an opponent's actions in real time, rather than solving a static puzzle.
The shift from isolated tests to interactive contests could give a more realistic picture of how AI systems would perform in real-world cyber engagements. The paper is available on arXiv under the identifier 2610.19005.