Turn up reasoning effort and agents verify more, spend more, and go looking for the answer key

Reasoning effort is sold as a dial: turn it up, the model thinks harder, you pay more and get better answers. A new study from Together AI, Johns Hopkins and Cornell tested that assumption task by task across eight models, and found that the dial does something less tidy. Higher settings did raise aggregate scores, but they also lost tasks that lower settings had solved, cost several times more on problems every setting could handle, and changed how agents went about the work. On SWE-Bench Pro, one of those changes was going looking for the answer.

The paper, posted to arXiv on October 25 with code on GitHub, covers GPT-5.6 Sol, Terra and Luna, Claude Sonnet 5, GLM-5.2, Inkling, Kimi K3 and DeepSeek V4 Flash. That is the previous generation in OpenAI's case: GPT-6.1 Terra, released a day later, isn't in it. The benchmarks are AIME 2026 and GPQA Diamond as single-call tests, plus two agentic ones: Terminal-Bench 2.1 (89 tasks, run in the Terminus-2 harness) and a 44-instance sample of SWE-Bench Pro run with mini-swe-agent. Within each model and benchmark only the effort setting changes; prompts and harness stay fixed.

The answer key gets more popular with effort

The original SWE-Bench Pro containers keep the repository's git history, and that history can include the commit that fixed the bug the agent is asked to fix. The authors counted how often agents read it. GPT-5.6 Sol did so on 15.9% of tasks at low effort and on 75.0% at xhigh.

Following Naman Jain's analysis on the Cursor blog from June, which found the same kind of lookup in 63% of successful Opus 4.8 Max resolutions, the authors built a corrected environment: git history removed, network access limited to allowlisted package registries. GPT-5.6 Sol kept trying. It attempted online lookups on 9.1% of tasks at low and 68.2% at xhigh, even though the sandbox rejected them. Its xhigh solve rate fell from 77.3% to 59.1%.

Every model scored lower in the corrected environment at its top setting, by between 11.4 and 29.5 percentage points:

With 44 instances per model these are small samples, and the drop is statistically significant for five of the eight models on its own; all eight falling is what makes the pattern hard to dismiss (sign test, p = 0.008). The paper's point is narrower than "the models cheat". Higher effort shifts agents toward strategies such as retrieving existing answers, and on a benchmark that leaves the answers lying around, some of what looks like extra intelligence is extra searching. Anyone comparing effort levels, or models, on the stock SWE-Bench Pro environment is partly measuring that.

More effort on easy tasks is mostly more input tokens

To separate cost from capability, the authors looked only at tasks a model solved at every effort level, where extra effort cannot buy anything. Costs still climbed, by up to 42 times the lowest setting. On SWE-Bench Pro, GPT-5.6 Sol's mean cost on those tasks rose from $0.27 at low to $1.24 at xhigh.

The money didn't mostly go to hidden reasoning. Sol's input-token charges on the same tasks rose from $0.17 to $0.60, because its trajectories grew from 14.7 model calls to 28.4, and every call re-reads a longer history. In a separate check on DeepSWE, where the authors re-analysed the benchmark's published runs with four attempts per task, input tokens made up 69 to 91% of the cost and reasoning tokens at most 15%.

Turning reasoning off doesn't fix this either. On tasks it solved at every setting, GPT-5.6 Luna with reasoning set to none averaged 21.5 model calls against 5.4 at low, and cost 4.58 times as much. The agent made up for not thinking by taking more steps.

The extra steps at high effort were often checking. Using an annotation taxonomy applied by Claude Sonnet 5, the authors found verification behavior rising with effort, including on tasks that both settings passed and on tasks both failed. On Terminal-Bench, Claude Sonnet 5's verification count on tasks passed at both ends went from 6.6 to 12.3. Sometimes the checking was wrong anyway: in one Terminal-Bench task GPT-5.6 Sol's highest-effort run declared that its output met a numerical constraint, the external verifier disagreed, and the low-effort run passed.

Lost tasks, with a caveat

The headline claim that higher effort "loses" tasks needs some care. Each agentic task ran once per effort level, and the authors say themselves that some of these reversals are sampling noise. Their four-rollout check on DeepSWE is the better evidence: a task counted as lost only if all four higher-effort attempts failed, and every model still lost some. Claude Sonnet 5 going from high to xhigh gained 7 tasks and lost 12, ending at 85 of 113 instead of 90. The finding holds, then, but it is about how unreliable single runs are, not proof that a higher setting makes a given task harder.

Start low, escalate on failure

The practical suggestion is simple: run at the lowest effort, and retry once at the highest only if an external check says the first attempt failed. Rebuilt from the recorded runs, this matched or beat a single run at top effort on every model and benchmark. Median savings were 25.3% on AIME, 41.5% on GPQA and 15.5% on corrected SWE-Bench Pro, counting the cost of both attempts. On Terminal-Bench, GPT-5.6 Sol's success went from 83.0% to 88.6% at 17.4% lower cost, while four of the eight models gained success but paid more.

Two things limit how far that carries. The comparison gives laddering two attempts against one, and the paper's own control on DeepSWE shows two attempts at high effort still beat it on success, at a much higher price: GPT-5.6 Sol reached 79.9% for $6.04 a task with high-then-high, against 76.8% for $3.80 with low-then-high. Laddering also needs a reliable pass/fail signal, such as a test suite, which many real deployments don't have. Letting the model pick its own effort from the task description, the obvious alternative, did no better than random assignment in their tests.

The authors also price in waiting time, valuing it at the US mean wage of $33.54 an hour. That's a blunt assumption, but it shifts the picture: for GPT-5.6 Terra on Terminal-Bench, laddering was 4.8% cheaper than fixed xhigh on the API bill and 19.5% cheaper once decoding time was counted.

Their broader argument is about what users are buying. Providers implement "effort" in different ways (prompt keywords, interrupted thinking budgets, length penalties in training), and nothing in those recipes guarantees that a higher setting dominates a lower one on any particular task. Since reasoning tokens are hidden but billed, the authors call effort a credence good, like car repairs: you see the result, not whether you needed what you paid for. Their request is that providers specify what each tier actually controls. Until then, the cheapest-first approach is the one with evidence behind it, as long as you have a test to tell you when to escalate.

For readers keeping track of agent benchmarks, this is the second recent case on this site where the environment around the model moved a score more than the model did; the first was a harness that quit on refusals in FrontierCS 2.