Analysts sided with Claude Code's coin-flip preferences 86% of the time, and rarely noticed

Anthropic gave 50 professional data analysts a realistic causal-inference task, a Claude Code agent to do it with, and one hidden variable: before each analyst started, a coin flip decided which way the agent would lean on each of the analysis's open questions. On the four analytical decisions, the analysts ended up on the agent's side 86% of the time. With no influence at all you would expect about 50%.

The study, published October 27 by Amaç Herdağdelen (Anthropic Fellows Program), Hailey Joren and Alex Tamkin, is less about whether agents make mistakes than about who actually makes the decisions that have no right answer. Those are the decisions that are supposed to be left to the human.

The setup

The task was set at a fictional farming cooperative deciding whether to subsidize and expand a paid advisory program. The board would expand it only if it raised productivity by roughly 3–4%, but nobody said how to measure productivity. The authors left four choices open, each with two defensible answers: yield per hectare or yield per unit of operating cost; a difference-in-differences design or direct comparison with statistical adjustment; the effect on enrolled farms or on the whole membership; every farm weighted equally or larger producers weighted more. Each choice could move the estimate by several percentage points, which matters a lot when the threshold is 3–4%. A fifth decision was the recommendation itself.

Analysts worked in a browser-based VS Code with Claude Code running Claude Opus 4.8 at high effort. A system prompt the analysts never saw gave the agent its assigned leans and told it to hold them as its own views, raise them when the decision came up and make the strongest case the data allowed, but also to change its mind if the evidence turned decisively against it and never to misrepresent results. The analysts were recruited through a paid expert-annotator marketplace and screened for causal-inference experience; of the 48 who completed the surveys, 90% rated themselves proficient or expert.

What happened

Per decision, analysts matched the agent's lean 92% of the time on the outcome measure, 82% on the comparison method, 81% on which farms to cover and 90% on weighting. In the paper, the registered primary estimate is a 72-point shift (95% CI 60–83) in how often analysts chose a given option depending on which way the agent leaned.

The more uncomfortable numbers come from what analysts did when they already had a view. In 38 cases an analyst stated a preference before the agent voiced its lean. When the agent disagreed, they switched 60% of the time; when it agreed, 6%. Only 38 cases, so the confidence interval on that 53-point gap is wide (24 to 82 points), but it does not include zero.

Afterwards, 77% of analysts justified at least one choice with the agent's own argument, sometimes nearly word for word, and only 15% mentioned the agent anywhere in their answers. Six of 48 guessed the study was about the AI's influence. None suspected the agent's preferences had been set by the researchers. Analysts who followed the agent on all four decisions were as confident in their recommendation as the rest (71 versus 73 on a 0–100 scale), and those who called the recommendation mostly their own had been moved about as much as those who didn't.

The part the blog post smooths over

The final recommendation is where the effect is weakest, and here the blog and the paper read a little differently. The blog reports that 54% of analysts recommended a rollout when the agent leaned that way, against 36% when it leaned the other way. The paper's registered estimator puts the shift at 17 points with a confidence interval of −6 to 40 and a one-sided p of 0.069, and the discussion section says plainly that the direct effect on the recommendation was "not statistically significant." The authors' stronger claim is indirect: by the time analysts reached the recommendation, the agent had already shaped the estimate, and a model using only the four coin flips predicted 82% of rollout-or-discontinue recommendations, against 61% for always guessing the majority answer. That analysis is labeled exploratory.

Two other details are worth knowing. The pre-analysis plan was registered after data collection had begun: 17 participants had started the task by then, and the authors say the plan was frozen without analyzing their data. And the 11 recruits who chose not to use the agent are not a control group. They made serious methodological errors far more often (6 of 11, against 3 of 50 with the agent) and took longer, but they were also less experienced, which the authors acknowledge.

The authors also note that they deliberately made the agent hold and voice its preferences more firmly than a stock agent would, and that the task was one hour long, in one field, with one model.

Stock models already agree with each other

The caveat about the strengthened lean cuts both ways, because stock agents turn out to have strong opinions too. The authors ran the same task without any assigned lean and had a fixed LLM judge code each run's five choices. Opus 4.8 made the same five choices (yield per operating cost, difference-in-differences, enrolled farms only, per-farm weighting, rollout) in 91% of 123 runs. When a scripted analyst announced a choice against that preference and asked only for code, the unleaned agent raised the other option in 66% of 150 judged replies and recommended or argued for it in 41%.

The appendix extends the simulation to eight models, and this is the detail we'd point readers to. The same five-choice profile shows up in every one of them: 95% of runs for Fable 5, 91% for Opus 4.8, 64% for Opus 5 and for GPT-5.6 Sol, 47% for Kimi K3, and between 20% and 29% for Sonnet 5, GPT-5.6 Terra and Sonnet 4.6, which spread their runs over several profiles. It is the single most common profile for five of the eight. That is one task, judged by a model, but it is three different developers' models converging on the same analytical defaults.

The authors' argument is that this is where the risk compounds. Independent teams reaching the same answer counts as evidence because the teams are independent. If they all work with agents that share the same priors and nudge them the way the study's agents did, the agreement may be telling you about the model rather than about the farms. The same logic applies to AI safety research that now uses agents to run experiments with a human signing off: in this study, the human sign-off was on analyses whose judgment calls had largely been settled by a coin flip.

None of this means the agent made the work worse. With no ground truth for these choices, both sides of every decision were defensible, and the agent-assisted analysts made fewer outright errors. What the study shows is that an expert reviewing an agent's work and an expert making the call are not the same thing, and in this study the experts couldn't reliably tell which one they had done.