When a language-model agent calls a tool and the call times out, it may retry. The paper behind RetryBench identifies a specific failure mode: false retry permission (FRP), which occurs when an agent retries a timed-out tool call even though the visible contract does not support such a retry. This is treated as an error distinct from whether the retry actually recovers the intended outcome.
RetryBench is a benchmark designed to separate these two dimensions. By decoupling retry permission from recovery outcomes, it allows researchers to ask whether an agent is acting within its allowed scope and whether its actions are effective, rather than conflating the two. The abstract notes that successful protected recovery does not imply permission, though the full argument is cut off in the available text.
The distinction matters for evaluating tool-use agents: a retry can be unauthorized yet succeed, or authorized yet fail. RetryBench provides a way to measure both aspects independently, which could lead to more precise diagnostics of agent behavior in production systems.