NVIDIA's "You don't need long context": Opus 4.5 and GPT 5.6 Sol keep their SWE-bench scores on 16K-48K windows, but the savings need new pricing

A team of NVIDIA researchers has put a blunt claim in the title of its new paper: "You don't need long context." In the arXiv preprint posted on November 3, Peter Belcak, Saurav Muralidharan, Pavlo Molchanov and nine co-authors show that Claude Opus 4.5 and GPT 5.6 Sol solve software-engineering, terminal and long-document tasks about as well when their working context is capped at 16K–48K tokens as when it is allowed to grow to 200K. In the most extreme setup the agent sees only one action per context and carries a short written summary of its state from one call to the next. On SWE-bench Verified, that still holds.

The accuracy results are solid. The cost story the paper builds on them depends on a price list that no API sells today.

Why short contexts are cheaper to serve

The paper starts with a measurement rather than a benchmark. Each new token is generated on top of everything already in the context, so the later it comes, the more it costs. The authors measured peak decode throughput for three open-weight models on rented H100s ($3.85 per GPU-hour) and converted it into the cost of a million output tokens. Serving at a 256K context window instead of 16K made each generated token 3.6 times more expensive for GLM 5.2, 5.6 times for Kimi K2.6 and 7 times for Qwen 3 Coder Next. On an NVIDIA DGX Spark desktop, several local coding models took roughly twice as long per token at the long end of the measured range as at the short end.

API customers don't see any of this, as the authors point out: retail prices are quoted per million tokens no matter how deep in the context those tokens sit.

Three ways to manage an agent's history

The experiments use mini-swe-agent, a minimal loop in which the model issues one Bash command at a time, and a generalized version of it the authors call mini-agent. They compare three regimes:

• Full-history accumulation: the usual approach, where every command and its output stay in the conversation.
• End-of-conversation compaction: when the context reaches a threshold, the model writes a compact summary of its state and work restarts in a fresh conversation with the task and that summary.
• Single-action compaction: the extreme version. The model reads its current state, takes one action, sees the result and writes an updated state for the next call.

With one action per context, the median peak context fell to 7–9K tokens and the largest single call to 15–37K, against 130–200K for the largest full-history calls. Across the 22 compaction settings the authors tried on long-horizon tasks, no more than five lost more than two points for either model. The losses that did show up were mostly in the tightest settings and on FRAMES, a multi-step search benchmark, where compaction cost 1.3–3.7 points.

The appendix on how to compact may be the most reusable part of the paper. The authors found that summaries made poor compactions: they spread attention evenly over the whole trace and over-weight whatever the agent repeated. What worked was a brief for the work still to do: a verbatim or near-verbatim copy of the task, what the agent is doing now and plans next, a plan, a list of requirements that define "done", and the artifacts or pointers to them that the remaining work needs. In the authors' words, the best compactions are "not narratives of the execution history but carefully cherry-picked briefs for further execution."

The cost column to read

The headline savings, 2.2x to 8.8x for Opus 4.5 and 2.4x to 5.0x for GPT 5.6 Sol on long-horizon tasks, are what the paper calls "estimated optimized retail cost". That is a hypothetical price model, described in Appendix B. The authors anchor the price at today's list price for the full window, assume that a 16K window would cost 20% of it (the average ratio from their open-weight throughput measurements), interpolate linearly in between, and apply that to the measured tokens.

The same tables also give the "nominal retail cost": the same token totals at the prices OpenAI and Anthropic actually charge. There the picture is mixed. On SWE-bench Verified, compaction made Opus 4.5 cheaper, from $420.00 for the full-history run to between $295.75 and $379.85, mainly because far fewer cached tokens were re-read. GPT 5.6 Sol went the other way: $167.68 for the baseline and $184.16 to $203.12 with compaction, since restarting conversations means more uncached input and more output tokens. On Terminal-Bench, the 48K setting raised Opus's bill from $87.80 to $114.75.

So the paper shows that short contexts are cheaper to serve, and that frontier models tolerate them. It doesn't show that a developer saves money by compacting today. That only happens if providers start pricing short windows lower, or if you serve the model yourself.

Long documents, read selectively

For long-input tasks the agent doesn't get the document in its prompt. It gets the document as a file and a Bash shell to read parts of it, with compaction as a fallback. The gains here are large: on RULER, mean recall rose from 62.5% to 95.7% for Opus 4.5 and from 65.5% to 96.4% for GPT 5.6 Sol; on AA-LCR, Opus went from 65.0% to 84.0% with a 16K window. Several of these benchmarks are needle-style tests, and the comparison is between pasting a whole document into the prompt and letting an agent search it, so it also says a good deal about how badly models still use very long prompts.

Where it breaks

The open-weight appendix is less tidy. At a 16K limit with compaction at 12K, every one of the five quantized local models lost accuracy on SWE-bench Verified. Qwen 3 Coder Next fell from 53.0% to 3.6%: according to the authors, its verbose trajectories left too little room for useful work between compactions. Qwen 3.6 27B and MiniMax M2.7 held up best. Frontier models seem to handle a tight window much better than smaller local ones, which matters because cheap local inference is one of the paper's stated motivations (the paper is part of NVIDIA's "Project Moonshot: Local Frontier Intelligence").

The closed models tested are also a generation old: Claude Opus 4.5 rather than Opus 5.5, and GPT 5.6 Sol rather than GPT-6.1 Sol.

Semantic state space models

From the one-action result the authors propose what they call semantic state space models (SSSMs): models post-trained to read a natural-language description of a task's state, take an action, record the outcome and rewrite the state for a fresh instance to continue. It is a state-space model in which the state is readable text rather than a hidden vector. If such models worked, the authors argue, context windows "could shrink drastically". That part is a proposal. No SSSM has been trained.

It is also a different answer to the problem we covered on November 3, when Prefrontal made one attention layer 72x faster at 10M tokens to make very long contexts affordable. NVIDIA's group suggests keeping the context short in the first place. Coding agents such as Claude Code and Codex already compact long sessions; what is new here is the evidence that much more aggressive compaction costs frontier models little accuracy.

The paper is careful about what it doesn't claim. It "does not rule out the existence of tasks for which long context is beneficial", and builds one itself: an agent that must recall the i-th of many coin tosses will eventually fail after lossy compaction. Its answer is to keep the full history on disk and let the agent search it, which makes the context short again. The conversation is still long. The agent just doesn't hold all of it at once.