Terminal-Bench 2.1: repairing ~30% of TB 2.0 tasks moves agent scores by up to 12 points — and most of a harness gap was the timeout

E. Kelly Buchanan, Crystal Zhou, Scott W. Linderman and Christopher Ré have a paper out today that treats an agent coding benchmark the way engineers treat software: something that accumulates defects after release and needs continuous repair.

Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks (arXiv 2611.08417) audits Terminal-Bench 2.0 — a hard CLI agent suite that already went through human review, LM-assisted auditing and red-teaming before launch — and ships the resulting repairs as Terminal-Bench 2.1. The authors call the backlog of task-side defects benchmark maintenance debt.

What broke

The audit found issues in 28 of 89 tasks, roughly a third of the benchmark. Defects fall into three overlapping categories (a task can have more than one):

• Instruction–verification mismatch (57% of the 28): the written task and the grader disagree. On query-optimize, for example, the verifier demanded byte-for-byte preservation of a database file that the instructions never mentioned.
• Task drift (43%): external state moved after release — package versions, APIs, web content — so a once-solvable task fails for reasons that have nothing to do with the agent.
• Environment issues (29%): CPU, memory or wall-clock budgets so tight that the oracle or the verifier itself cannot run reliably.

Fourteen of the 28 repairs are new to this audit; the rest consolidate earlier public reports, including Z.AI's Terminal-Bench 2.0 Verified set.

Scores move; the ranking mostly doesn't

Re-running 19 agents on both versions, accuracy rises by more than 10 percentage points in aggregate terms the authors highlight, with individual agents shifting by up to 12 points and moving up to three leaderboard ranks. The overall ordering stays highly correlated (Spearman ρ = 0.958).

The biggest lifts sit on instruction–verification mismatches: five tasks that had pass rates of at most 15% on TB 2.0 jump above 50% once the verifier stops rejecting valid solutions. That is the useful distinction. Some agents were already solving the task; the old grader was not recognising it.

The harness gap was mostly a clock

The paper's second claim is about resources, not just buggy tasks. At the default wall-clock budget, more than half of failed TB 2.1 trials are timeouts (up to about 68% under Claude Code). Extending the timeout to 32× closes a roughly 20-point Codex-versus-Terminus-2 gap on GPT 5.4 and shrinks a 30-point gap on GPT 5.4 mini to about 5 points.

In other words, a fixed-budget harness comparison can look like a capability difference when it is largely a speed difference under a shared timeout. The authors treat the resource budget as part of the measurement specification, not an afterthought.

Continuous validation, not a one-shot filter

The workflow is deliberately agent-assisted and maintainer-gated: an evidence collector pulls task files, traces and community reports; an auditor proposes a cited repair under a short rubric; /oracle rebuilds the task and runs the reference solution; humans decide what ships. Managed coding assistants (Claude Code with Opus 4.6, Codex with GPT 5.4) agree on the primary defect label at κ = 0.80 across all 89 tasks. Lower-cost API auditors are usable for triage but weaker on attribution, especially task drift — hence the deterministic oracle check.

That is a different answer from the usual "Verified" subset approach (filter the broken tasks and shrink the set). TB 2.1 repairs in place, documents difficulty changes, and versions the release.

Where to look

The repaired tasks live at harbor-framework/terminal-bench-2-1; the discussion of each fix is in PR #53 on the TB 2.0 repo. Buchanan's project page summarises the numbers and hosts a snapshot of the workflow.

One caveat for readers comparing this paper to Artificial Analysis or vendor cards: those often cite Terminal-Bench 4.0, a later generation. The audit here is specifically of Terminal-Bench 2.0 → 2.1. The maintenance lesson travels; the score deltas do not automatically transfer to other Terminal-Bench versions.