Jailbreak research typically reports an attack success rate: the share of responses that deliver the harmful request. That number is usually read as a property of the target model being attacked. A new arXiv preprint challenges this assumption by holding the target model, the prompts, and the responses fixed, and changing only the LLM grader used to judge whether a response is harmful.
The authors find that the choice of grader matters. Different LLM graders produce different success rates for the exact same set of responses, meaning the reported number is partly an artifact of the evaluation setup. The paper's title points to a "missing compliance class" — a category of responses that are neither clearly compliant nor clearly non-compliant — which graders appear to collapse, distorting the final success rate.
As a practical remedy, the researchers offer a "free check": a simple diagnostic that lets other researchers test whether their own grader is reliable. The work is a methodological caution for the jailbreak literature, suggesting that attack success rates should be interpreted with attention to the grader as well as the target model.