OpenDS-World exposes the time cost of failed tool calls

Failed tool calls are a small share of research agents’ actions but can consume most of their tool-execution time. In OpenDS-World, researchers report that failures accounted for 2.83%–9.42% of calls across the models tested, yet consumed 19.27%–61.90% of the time spent executing tools.

For GPT-6 Astra, the figures were 6.34% of calls and 42.57% of tool-execution time. Those percentages concern tool execution, rather than the agent’s entire runtime. A low error count can therefore conceal an expensive failure path.

The OpenDS-World benchmark contains 200 research tasks across eight domains. Agents must search the live web, find relevant evidence and piece together an answer. Twelve models were tested under the same tool setup and search budget. The best scored 39.0%, although these tasks are considerably harder than everyday web searches.

Each task has a one-hour deadline and a maximum of 100 tool calls, according to the harness documentation. The hour includes both model processing and tool execution. Failed calls count toward the limit, and the recorded traces show how much time agents spent on tools, including unsuccessful attempts.

Scoring introduces another boundary. The evaluation guide uses a language-model judge to compare answers with references and counts failed executions as incorrect. Its results measure the agent and its execution setup together.

The study does not establish that removing tool failures would produce a corresponding accuracy gain. It does identify a useful diagnostic: count the seconds lost to failed calls alongside the calls themselves.