Artificial Analysis' new index puts open models 12 points behind, with the widest gaps on public benchmarks

Artificial Analysis has rebuilt its Intelligence Index, the composite score many developers use as a quick read on where models stand, and the headline result is that open-weights models now look much further behind. In version 5, published on October 28, the best open-weights model trails the leader by 12 points. In v4.3 the gap was 3.

It is tempting to read that as the payoff of the index's other big change. For the first time, most of it (60%) rests on private test sets or held-out answers, which is exactly what you would build if you suspected public benchmarks were flattering some models. Ethan Mollick put it that way: Artificial Analysis "finally moves to closed benchmarks" and finds open models "falling further behind". The per-benchmark numbers tell a more complicated story.

What changed in the index

v5 has nine evaluations in four equally weighted groups: knowledge work, coding, science and general. According to the methodology page, the new or upgraded entries are:

• FrontierCode 2.0 (15%), Cognition's private benchmark of 132 tasks in real open-source repositories, graded on whether a maintainer would merge the change; failing any blocking check scores zero.
• Terminal-Bench-Science 0.1 (15%), 70 expert-reviewed research tasks run in a terminal, where a task passes only if all its tests pass.
• HLE-Diamond (10%), a cleaned 1,000-question version of Humanity's Last Exam, replacing the original.
• CritPt v2, hard subset (10%), 40 research-level physics problems: the full set of 70 minus the 30 that seven frontier models solved in all five attempts at launch.
• AA-Omniscience v1.1 (15%), the knowledge and hallucination test with saturated and public questions removed.

GDPval-AA, SciCode and the long-context test AA-LCR are out. AA-Briefcase, AutomationBench-AA, GDP.pdf and Terminal-Bench 4.0 stay. The 60% figure checks out against the table: AA-Briefcase, AutomationBench-AA, FrontierCode 2.0, the CritPt subset and AA-Omniscience are marked private, and their weights add up to exactly 60.

At the top, Claude Opus 5.5 (max effort) scores 58, GPT-6 Astra (max) 57 and GPT-6.1 Sol (max) 55. Gemini 4 Argon (high) is next at 50. The best open-weights entry is Kimi K3.1 Pro at 46, with an asterisk Artificial Analysis supplies itself: its weights are not out yet. Moonshot has promised them by November 4. Among models you can actually download, DeepSeek V4.1 Flash 1017 and GLM-5.3 (both at max) lead with 32, below Claude Haiku 5.5 (max) at 36.

Where the gap actually is

Artificial Analysis says the gap is widest on HLE-Diamond and Terminal-Bench-Science. Kimi K3.1 Pro scores 37% and 34% there, against 68% for Claude Opus 5.5 and 63% for GPT-6 Astra, the respective leaders. According to the methodology table, neither of those two is private: HLE-Diamond's questions are published on Hugging Face and Terminal-Bench-Science's tasks are on GitHub.

The private evaluations aren't kind to open models either. On the CritPt hard subset, Kimi K3.1 Pro's 45% is 27 points behind GPT-6 Astra's 72%. On FrontierCode 2.0, the best open-weights model, DeepSeek V4.1 Flash 1017, reaches 28% against Opus 5.5's 46%. And on Terminal-Bench 4.0, which is public, an open-weights model is in first place: Kimi K3.1 Pro, at 67%. That is close to the 65.7% Moonshot reported for it, so here a vendor number held up in someone else's harness.

So the split doesn't run along private and public lines. A better predictor is how new and how hard the benchmark is. Cognition's FrontierCode 2.0 announcement shows this within one benchmark family. On FrontierCode 1.1, Kimi K3 scored 44.2% to Opus 5.5's 54.6%, a gap of about 10 points. On 2.0, which Cognition says has larger tasks and stricter merge criteria, Kimi K3 scores 17.7% and Opus 5.5 45.5%, a gap of about 28. DeepSeek V4.1 Flash goes from 43.9% to 19.3%. Cognition adds that the results show "significant room for improvement for open source models, even more so for western OSS models". The split isn't strictly open versus closed, though. Opus 5.5 and GPT-6 Astra lose 9 and 10 points between versions, while Grok 4.7 (47.6% to 28.6%) and Gemini 4 Argon (47.5% to 26.3%) fall almost as far as the open models. The harder benchmark separates the top two from everyone else, and the open models are among everyone else.

None of this proves that the old gap of 3 points was an illusion, or that the new gap of 12 is the true one. It does suggest that the move to harder, longer tasks, more than secrecy as such, is what pulls the field apart.

A more expensive index to run

The other consequence is cost. v5 asks for much more work per task: the median output per Intelligence Index task rose from 35k tokens to 59k, and every model measured produced more than under v4.3. The four agentic evaluations carry 55% of the weight but account for a median 88% of output tokens, and Terminal-Bench-Science alone takes about a third.

That makes effort settings matter more. Claude Opus 5.5 scores 35 at low effort and 58 at max, at $0.98 and $10.04 per task respectively. GPT-6.1 Sol's spread between low and max doubled from 10 points to 20. Among the ten highest-scoring models, GPT-6.1 Terra (max) is the cheapest at $0.96 per task and Claude Fable 5.1 (max) the most expensive at $13.04. A single index number now hides a lot of choice about how hard the model was asked to think, and anyone comparing a score to their own bill should check which effort level it came from.

The private sets come with a trade-off Artificial Analysis can't avoid. They are harder to train on, and also impossible for anyone else to rerun. The company has released a recalibrated public subset of AA-Omniscience v1.1 on Hugging Face, but for FrontierCode 2.0, AA-Briefcase and the CritPt subset, readers have to trust the grader. As our FrontierCS 2 story showed, an agentic score measures a model together with the harness running it; here that is mostly Artificial Analysis' own Stirrup harness and mini-swe-agent.

The open-weights figure also comes with a timing caveat. The model that defines it, Kimi K3.1 Pro, can't be downloaded yet; if the weights ship as promised, the 12-point gap becomes a real one for self-hosters. If they don't, the honest number for open models you can run today is 26.