A new preprint introduces Lumen-bench, a benchmark designed to probe safety in large language models that can call external tools. The authors frame the core issue as a "say-do" problem: in plain conversation, a refusal is bounded by what the model says, but once the model has executable tools, what it does can diverge from what it says.

The benchmark targets cross-lingual safety gaps, asking whether models that appear safe in one language remain safe in another when tool use is available. This matters because tool-calling models can act on the world, not just produce text, so a refusal in words does not guarantee a refusal in behavior.

As the only source reviewed here, the abstract does not report specific results or compare models. Instead, it lays out a measurement problem and a benchmark for studying it.