A new paper on arXiv tackles a core problem in evaluating tool-using agents: how to know when a robustness test is actually testing the same task as the original. Many tests change the agent's environment—for example, by hiding a hostile note in an invoice—and then judge the agent by what it does. The paper points out that the result tells us about the original task only if certain conditions hold, though the abstract is truncated before those conditions are fully spelled out.
The paper's title suggests a solution: "executable contracts" for robustness evaluation. These contracts would presumably specify, in a runnable form, what counts as the same task across environmental changes. This would give researchers a more principled way to design and interpret robustness tests, rather than relying on intuitive judgments about whether a modified scenario still represents the original problem.
Because the source is a single abstract, the details of the proposed contracts are not available. The main contribution, as stated, is the framing of the problem and the introduction of executable contracts as a formal tool for addressing it.