Safety-aligned large language models are expected to refuse harmful requests while remaining useful for benign ones. But a new paper on arXiv argues that many evaluations only score the final response, which can conflate a model's understanding of harmful intent with its actual refusal behavior. The authors introduce the concept of 'recoverability-controlled safety miscalibration' to describe this issue.

According to the abstract, when evaluators look only at the final output, they may not be able to tell whether a model correctly understood the request's intent but failed to refuse, or whether it never understood the intent in the first place. This miscalibration can lead to misleading safety assessments.

While the full paper is not yet available, the abstract suggests that the proposed framework aims to separate intent understanding from refusal behavior, potentially offering a more nuanced way to evaluate safety-aligned models.