A new paper on arXiv examines how AI agents that act through tools are supervised by safety guards, and how those guards are themselves evaluated. The authors argue that reliable evaluation is essential, especially for sensitive actions, but the data used for such evaluations can be flawed.
The problem, as described in the abstract, is that evaluation datasets may contain cues that make labels predictable without requiring the guard to genuinely detect unsafe behavior. This "shortcut sensitivity" means a guard could appear effective in testing while failing in real-world deployment.
The paper is a single source, so there are no contrasting findings to compare. Its contribution is to highlight this specific failure mode in agent-safety evaluations and to call attention to the need for evaluation data that does not rely on superficial patterns.