A harness that quit on refusals cost Gemini 4 Argon seven tasks on FrontierCS 2

FrontierCS 2 launched last week with a demanding design. Each of its 48 tasks takes a published computer science paper, deletes the paper's key contribution from the authors' own code, and gives an agent five hours to write a replacement. A run passes only if the final submission beats the original implementation on every metric and every hidden workload.

The paper pairs each model with one harness, and it says the harness "is part of what we measure". Most models ran in their vendor's own tool: Claude Code, Codex CLI, Kimi Code, ZCode. Gemini 4 Argon ran in Pi, a third-party harness. That pairing mattered more than it seemed.

Wenhao Chai, one of the benchmark's maintainers, says an audit of the traces found that Pi "just ends the run whenever it hits a fallback". The leaderboard's notes are more specific. In 10 of Argon's 48 runs, the model provider's content filter refused a request and Pi stopped the run. Every run without a passing final submission counts as a failure, so those refusals counted as failed tasks.

Rerun through Google's Antigravity CLI, which retries refused requests, Argon passed 26 of 48 tasks (54.2%) instead of 19 (39.6%). According to the Kaggle leaderboard, that moves it past GPT-6 Astra (43.8%, Codex CLI) and just behind Claude Opus 5.5 (58.3%, Claude Code). The Pi result is still listed, but as a hollow marker outside the ranking.

The rerun was not free. Mean cost per run rose from $10.44 to $19.52, which puts Argon level with Opus 5.5 ($19.98). Retries are not a cure-all either. The site notes that in one cryptography task Antigravity's retries ran out and the run ended. In the CREST task, context summaries that drew refusals were kept in truncated form.

Argon is not the only model whose run was changed by a refusal. The same notes say that in the Progent task, a refusal switched the rest of Opus 5.5's Claude Code run to Claude Opus 4.8.

None of this makes the benchmark less useful. It does show what a pass rate on a long agent benchmark measures: a model, a harness, and the provider's safety layer between them. The same model scored nearly 15 points apart in two harnesses, and the documented difference is how each one handled refusals. FrontierCS published the refusals and kept both Argon results, which is the right way to handle it. Kaggle labels the scores as externally sourced and says it is partnering with the authors to reproduce and verify them.