Jailbreak attacks against large language models are usually measured by attack success rate, or ASR. That metric relies on an evaluator correctly deciding whether a model's response actually fulfills a prohibited request. According to a new preprint on arXiv, existing evaluators are prone to error, sometimes mistaking refusals or unrelated outputs for successful jailbreaks.

The authors propose a "semantic reconstruction game" as an alternative way to evaluate responses. Rather than relying on a direct classifier, the game likely involves reconstructing the original request from the response and comparing it against the prohibited intent, though the abstract is truncated and full details are not available in the source.

The paper, arXiv:2610.19306v1, argues this approach can make jailbreak evaluation more reliable. The source does not provide experimental results or comparisons, so the claimed improvement is not yet verified in the abstract alone.