Attack success rate (ASR) has long been the standard for measuring how well jailbreaks work against large language models. But according to a new arXiv paper, ASR has a blind spot: it does not account for the repeated calls that many iterative jailbreak strategies make to the victim model.
The authors argue that each additional call is not free. In a deployed system, every extra request gives the defender another opportunity to spot and stop the attack. A metric that ignores this cost can overstate the practical threat of a jailbreak that only succeeds after many attempts.
The paper therefore proposes a cost-aware metric that factors in the number of calls needed to achieve a successful jailbreak. This shifts the evaluation from a simple yes/no outcome to a more realistic measure of the effort and risk involved in an attack.