When researchers report how well a tool-using AI agent performs, the number usually reflects more than the model itself. The harness that runs the agent — with its limits on time, memory, or number of tool calls — can inflate or suppress scores. That makes it hard to know whether a result comes from a better model or simply a more generous setup.
ToolBottleneckBench, a new benchmark from an arXiv preprint, directly addresses this problem. It evaluates tool agents under controlled pressures while keeping the harness budget fixed. By varying what the agent has to cope with — for example, harder tool descriptions, more competing calls, or stricter resource limits — the benchmark aims to pinpoint where performance degrades and why.
The significance is practical: separating model capability from harness constraints lets developers decide where to invest. If a bottleneck is in the harness, upgrading the model may not help. If it is in the model, throwing more budget at the harness will not fix it. The authors argue that the benchmark offers a more honest way to compare tool agents, though the preprint has not yet been peer-reviewed.