A new benchmark called HardEval aims to evaluate AI systems on security hardening and compliance remediation. The work, posted on arXiv, focuses on Security Technical Implementation Guide (STIG) controls, which are a demanding operating-system administration task. Each control is expressed as a natural-language security requirement that must be translated into a precise and persistent system configuration.

HardEval is described as execution-verified, meaning that AI-generated actions are checked against actual system outcomes rather than just textual similarity. This makes it a practical test of whether an AI agent can correctly implement STIG requirements on a live system.

The paper highlights the challenge of moving from natural language to concrete, durable changes in an OS environment. Because the source is a single arXiv announcement, no independent comparisons or additional context are available yet.