Agent Initiative Benchmark: all ten of Gemini 4 Argon's "correct silences" were runs that hit the 180-second clock

Deepnote's Agent Initiative Benchmark, released on November 13, asks a question most agent benchmarks skip: can a model tell when not to say anything? Twelve models sit in the Slack of PageMantle, a synthetic software company with records spread across eleven simulated services (Slack, Notion, Linear, HubSpot, Stripe, PostHog, BigQuery, Snowflake and others). In each of 50 situations the agent can investigate, then post in a channel, reply in a thread, send a DM, or stay quiet. Forty situations need help; ten don't. Two LLM judges score every run from 0 to 10.

The headline table puts GPT-6 Astra first at 8.97, GPT-6.1 Sol second at 8.65 for $0.11 a task, and Claude Fable 5.2 third at 8.17 for $1.87. Gemini 4 Argon (5.60) and Grok 4.7 (4.51) finish last. Deepnote's most quoted finding is that models are "good at helping and bad at not helping": a message went out 9 times in 10 when help was needed, but was held back only 6 times in 10 when silence was right.

That finding holds. One cell in the table supporting it doesn't mean what it appears to.

Gemini's perfect restraint is ten timeouts

In the "stayed quiet" column, the model with the best record is not Astra or Opus at 8/10. It is Gemini 4 Argon at 10/10. The same table, plus two facts from Deepnote's own write-up, shows where that score comes from.

Every run has a 180-second budget of model response time. Deepnote reports that Gemini hit that limit before sending any message in 27 of its 50 runs. It also reports that Gemini helped in 23 of the 40 tasks that needed help, so it missed 17. A timeout on a help task is a miss, so at most 17 of the 27 timeouts can fall on help tasks. The other ten or more have to be on silence tasks, and there are only ten of those.

So every one of Gemini's ten "correct silences" was a run that ran out of time while still investigating. The benchmark's judge prompt, published in the repository, explains why they count anyway: a stop caused by the response budget "is graded exactly like finish_task", with no timeout penalty, and "correct silence receives 10."

That also explains Gemini's eleven perfect scores, a number that otherwise looks out of place for a model in eleventh place. Ten of them are those timed-out silences. Take the 100 points from those runs out of Gemini's total (5.60 × 50 = 280) and its average on the 40 tasks that needed help is 4.50, our arithmetic from Deepnote's published figures.

Grok 4.7 has the same problem on a smaller scale. It timed out in 18 runs, missed 16 help tasks and stayed quiet on 6 of 10 silence tasks, so at least two of those six silences, and possibly all of them, were also clock-outs.

None of this is hidden. Deepnote writes that most missed messages "were timeouts, not deliberate choices to stay silent". But the same logic applies to the silence column, and there the table presents Gemini as the most restrained model in the test. A dashboard reader comparing Gemini's 10/10 with Opus 5.5's 8/10 would be comparing a clock with a decision.

Removing Gemini makes the headline slightly worse, not better: the other eleven models held back in 65 of 110 silence situations, 59%.

Ten tasks can't rank restraint

The bigger limitation is the size of the silence panel. Each model faced ten silence tasks, so each "stayed quiet" figure is a count out of ten. Our Fisher exact test puts the difference between 8/10 and 6/10 at p = 0.63; even 8/10 against 4/10 gives p = 0.17. The 95% Wilson interval for 8/10 runs from 0.49 to 0.94, for 6/10 from 0.31 to 0.83.

That matters because the write-up draws model-level conclusions from those counts, for example that Opus 5.5 is "among the best at staying quiet" while Fable 5.2 is "mid-table at holding back". This panel can't separate them. The aggregate claim, roughly 62% across 120 silence runs against 90% across 480 help runs, is well supported; the per-model restraint ranking is not.

To its credit, Deepnote is careful with the overall scores. It reports task-bootstrap intervals and says plainly that Astra (8.30–9.52) and Sol (7.88–9.31) are not separated by this task set. Fable 5.2's 8.17 sits inside Sol's interval too.

The clock includes the network

The 180-second budget counts "awaited model API response time, including provider and network waits", according to the runtime message in the repository; tool execution is excluded. Gemini's median run took exactly 3 minutes 0 seconds, and Grok's 90th percentile hit the same wall. With 34.5 and 35.5 tool calls per task on average, both models clearly spent the budget investigating, and Deepnote's advice to give a harness "a point at which it must commit" follows from that.

But part of what the bottom two rows measure is how fast each provider's API answered on the day of the run, not only how the model reasons. We have seen the pattern before: last month a harness that quit on refusals cost Gemini 4 Argon seven tasks on FrontierCS 2. Deepnote says this version "compares models, not harnesses" and that a different prompt or tool policy "will move every number here". For Gemini and Grok, the harness's clock is most of the story.

What is open, and what isn't

The environment is the most useful part of the release. The simulated service APIs and judging pipeline are on GitHub under Apache 2.0 (deepnote/agent-initiative-benchmark), and the PageMantle data, about 6 GB to download, is on Hugging Face as deepnote/pagemantle-company-dataset under CC BY 4.0. The repository ships eight runnable example tasks, one per task type, and says those "are not a score for the full benchmark". The other 42 tasks and the traces of the 600 leaderboard runs are not in the repository, and the write-up doesn't name the two judge models. The repo's sample configuration pairs a Responses-API judge with an Anthropic one; if the leaderboard used something similar, judges from the same two labs whose models top the table would be worth knowing about. Until the full task set appears, the per-task claims can't be checked independently, only the arithmetic.

The cost comparison survives that arithmetic. Deepnote's monthly figures, $281 on Sol and $4,684 on Fable 5.2, both imply about 2,500 tasks a month, and Luna ($0.007) against Fable 5.1 ($3.02) is indeed a 430× price gap for 1.1 points of score. For a team picking a model for an always-on Slack agent, that part of the table is probably the most actionable.