SO-Arena benchmark: OpenAI's Decisions API scores 26 points below GPT-6 Luna, as Gemini Decision 1 arrives untested

Google launched Gemini Decision 1 on November 11, and with it the decision model became a product category with three big-name entries. TypeSafe's Jev started it in September, OpenAI followed with its Decisions API on gpt-6-luna (public beta October 6, generally available October 22), and now Google has gemini-decision-1. All three do the same unusual thing: they don't write text. You send an input and a batch of typed questions (yes/no, pick one of N, grade on a rubric), and the model returns probabilities from a single forward pass, billed on input only.

The same day, the first broad independent benchmark of this category appeared. "SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models" (arXiv:2611.15777), by Lichen Li, Ziping Dong, Sicheng Feng, Qi Li and Xinchao Wang, tests 21 models, including Jev API 1.14 and OpenAI Decisions. Its most useful result is one the abstract doesn't mention: OpenAI's Decisions API scores 26 points below the GPT-6 Luna model it is built on. We reran the leaderboard from the authors' published per-task records to check that, and found that the paper's headline reliability finding rests on a two-hundredths-of-a-point margin.

Three APIs, one shape

The three products are close to interchangeable at the API level. Google even reuses Jev's coined term "noul" for a yes/no question, as Simon Willison pointed out.

Gemini's pitch rests on modality and scale: one shared input of up to 262,144 tokens, up to 1,000 questions over it, and 150 to 250 ms end to end, according to Google's announcement by Min Choi and Felix Yu of Google DeepMind. Google also says Gemini Decision 1 "leads across the suite, with the lowest calibration error" on its own benchmarks. It publishes no table an outsider can check, and SO-Arena, finished before the launch, doesn't include it. For now that claim is Google's alone.

One design difference is visible without any benchmark. Gemini's documentation says a yes/no answer is a "calibrated probability from 0.01 to 0.99", so it never claims certainty. In Willison's quick test with a video, Gemini answered "Does this video have a robot in it?" with 0.92; in his tests, Jev and OpenAI's API answered yes/no questions with 1.0 or 0.0. Whether 0.92 is better calibrated than 1.0 is exactly what a benchmark should measure; nobody has measured it for Gemini yet.

What SO-Arena measured

SO-Arena covers 11 domains, from sentiment and knowledge questions to math, code understanding, tool use and multi-step web tasks in sandboxes. Its leaderboard averages the 11 domains equally, using only the hardest tier of each graded domain: 1,232 tasks. It also measures reliability two ways: whether a model keeps its answer when only the order of the options changes, and whether it notices when none of the options is correct.

The comparison includes two ordinary "System Two" models that reason before answering, GPT-6 Sol and GPT-6 Luna, both through OpenAI's Responses API at medium effort. They lead: Sol 82.3, Luna 79.7. The best decision models are about ten points behind. The authors split the gap by domain. On knowledge, sentiment and retrieval questions, decision models are roughly as good as Luna. On math, scientific reasoning and code understanding they fall far behind, and there thinking is what helps. StartLux Auto, a 4B model that can switch reasoning on, goes from 50.8 to 92.5 in math when it always thinks. The price is time: StartLux Auto averages 3,084 ms per task, against 35 ms for StartLux Instant.

OpenAI's Decisions API vs GPT-6 Luna

Here is the part worth knowing if you are deciding whether to move a classifier onto one of these APIs. The paper's model table lists both OpenAI entries with the same identifier, gpt-6-luna: one called through the Responses API, the other through the Decisions API. Their leaderboard scores:

We recomputed these from per_task_credits.jsonl.gz in the authors' SO-Arena dataset, using the panel rule from the scoring script in holdrain/SO-Arena. Every score matched the paper. A bootstrap over tasks (resampling within each domain, 1,000 to 2,000 rounds) puts the gap between Luna and its Decisions version at 26.1 points, with a 95% interval of 23.1 to 29.0. That is not noise. OpenAI Decisions also trails Jev API 1.14 by 6.8 points (interval 3.1 to 10.6) and StartLux Instant, a 4B open-weight model, by 5.4 (1.8 to 9.1).

The domain scores show where the points go:

So the pattern follows the rest of the paper: on questions you can answer at a glance, the Decisions API is close to Luna; on questions that need working out, it isn't, because it doesn't work anything out. That is the deal OpenAI describes. Its GA announcement promises answers "up to 10x faster than GPT‑6 Luna through the Responses API", and SO-Arena's timings agree almost exactly: 302 ms against 2,698 ms, about nine times faster. OpenAI's customer quotes in the same announcement (Box, Ramp, Sierra) all say accuracy held up. They were also describing routing and classification jobs, the domains where SO-Arena finds the smallest gap.

Two weaknesses are specific to OpenAI's API, not the category. Its order stability is 0.68, one of the four lowest among the 18 decision models: shuffle the same options and it changes its answer noticeably more often than Jev (0.89) or GLiDE (0.94). And it refuses. The authors count four refused static tasks and twelve refused tool-use episodes, each scored zero. One example in the paper is a Safety-domain item whose context combined instructions to disregard safety restrictions with a harmful request, while the question only asked for a classification. Another is a web form field asking for an exact value from the page. The API returned type: refusal with no explanation. If your pipeline classifies harmful content, which is a common use for exactly this kind of model, that is worth testing before you rely on it.

Two caveats keep the gap in proportion. The leaderboard deliberately uses only the hardest tier. Averaged over all 2,919 tasks in the release, without domain weighting, the scores are 85.6 for Luna and 70.3 for OpenAI Decisions, a 15-point gap rather than 26. And Task Completion Verification has only 20 hard tasks while counting for a full eleventh of the score: Luna got 20 of 20, OpenAI Decisions 8, Jev API 1.14 just 4.

The headline finding rests on a tie

SO-Arena's abstract says that "the best-performing System One model ranks only seventh in reliability". The reliability values in the authors' results/paper_values.json bear out the ranking: GLiDE's reliability score is 77.6, seventh of 18. But GLiDE is "best-performing" by 70.212 points against 70.191 for KnowLine-31B, a difference of 0.021. In our bootstrap, KnowLine came out ahead in 49% of resamples, and the 95% interval for the difference ran from minus 2.5 to plus 2.4 points. KnowLine's reliability score is 81.0, fourth of 18.

So the paper's broader point survives: capability and reliability rankings really do diverge. The two most reliable decision models, JevAny-27B (84.1) and Jev API 1.14 (83.5), are only eighth and sixth in the 18-model performance ranking. But "the best model is only seventh" is a statement about a coin flip between two models.

Two more papers in the same week

The category is also getting research of its own. "Actor, Judge, or Teacher? Understanding Decision Models in Long-Horizon Agents" (arXiv:2611.17167), from Haochen Liu, Yongliang Shen and colleagues at Zhejiang University, finds that decision models make poor agents but good graders. As actors in ALFWorld, all six models they tested get pulled toward an easier subgoal instead of the one that has to come first. Used after the fact to score each action of an episode, Jev raised a Qwen2.5-1.5B policy trained with GRPO from 72.8% to 82.9% success on ALFWorld and from 56.8% to 71.4% on WebShop. That fits SO-Arena: judging a finished trajectory is a look-at-it question, and planning ahead is the multi-step kind of task these models are weak at.

"From Large Language Models to Efficient System One Models via Adaptive Computation" (arXiv:2611.16321) by Minh-Anh Nguyen looks inside them. Turning a language model into a decision model, he writes, typically means replacing its text head with a decision head and keeping the whole backbone, so every decision pays for every layer. Nguyen reports that the decision information sits in the intermediate layers, and that the final six layers of the models he studied can be removed without loss. His NestOne method trains early exits so one model can stop at whatever depth a question needs.

What to take from it

Decision APIs are cheap and fast because they skip the reasoning, and the first independent numbers say that is where their accuracy goes too. For routing, tagging, sentiment and other questions that need no working out, they come close to a reasoning model at a ninth of the latency. Anything that needs a calculation or a plan still needs a model that thinks. Among the APIs, OpenAI's is the clearest example of the trade: the same model name, 26 points lower on SO-Arena's hard panel, and nine times faster.

Gemini Decision 1 is the one entry nobody outside Google has measured. Its multimodal input and 1,000-question requests are genuinely new for the category. Its calibration claim is the most testable thing in the launch, and it's the one to watch once SO-Arena or anyone else adds it to the table.