MCPHallu: agents on live MCP servers score 0.64 on average, and Unreachable Goal is where they pretend hardest

Yizhen Jiang, Haowei Guo, Tianyi Bai and co-authors (Peking University, Zhongguancun Academy, Meituan, HKUST) have released MCPHallu — MCPHallu: Benchmarking Reasoning, Execution, and Memory Hallucinations in MCP Agents (arXiv 2611.07689) — a benchmark that asks not whether an MCP agent finishes a task, but which kind of hallucination it tends to produce when it fails.

That framing matters for anyone wiring agents to real Model Context Protocol servers. Existing MCP suites mostly score completion. MCPHallu builds each of its 358 tasks around a structural trigger that makes one of four failure modes the dominant risk: Branch Collapse (wrong conditional branch after a live tool result), Unreachable Goal (no available tool can do what was asked), Tool Misuse (picking a lookalike tool with the wrong semantics), and Context Forgetting (dropping a value retrieved earlier once distractors pile up). The suite runs against 36 production MCP servers in isolated containers — search, analytics, productivity, finance, coding — rather than mock APIs.

What the leaderboard actually says

Fourteen frontier models from eight vendors ran 5,012 trajectories. Mean score is 0.642 on a [0, 1] scale; the best reported overall is Claude Opus 4.6 at 0.728. No model clears that bar by much, and the ranking is not stable across subtypes: a model's place shifts by about 6.9 of 13 positions on average when you change the failure mode. Branch Collapse and Unreachable Goal are essentially uncorrelated across models (r = −0.12). The overall winner is only mid-pack on Branch Collapse. Collapsing everything into one MCP score, the authors argue, hides the capability you care about for a given deployment.

The hard case is Unreachable Goal. Average score falls to 0.458, with only 23.2% of runs passing. Agents often keep calling tools — or write completion language such as "saved" / "created" — instead of saying the goal is infeasible. On those tasks, runs that make zero tool calls pass 65% of the time; once the agent has issued four or more calls, the pass rate in the authors' binning is 0%. Extra tool use here behaves less like exploration and more like a hallucination amplifier.

Tool Misuse has a similar discipline problem. Calling only the correct tool yields an 84.7% pass rate; mixing it with curated "confusion" tools drops that to 32.3%, even though the right tool was still in the trajectory. Discovery is not the bottleneck — avoiding contaminated intermediate state is.

Scoring is mostly deterministic: live reference scripts or post-execution state assertions for Branch Collapse and Context Forgetting, trajectory-structure checks for Unreachable Goal and Tool Misuse. An LLM judge (GPT-4o in the main tables) contributes only 20–30% of each subtype score; swapping the judge for Claude Sonnet 4.5 moves scores by at most 0.016 on average.

What to keep in mind

The evaluated lineup mixes generations (GPT-4o and GPT-5-chat alongside Claude 4.5/4.6, Gemini 3 Flash, DeepSeek-V4, Qwen3.5-397B, Kimi-K2.6, and others). Treat the absolute numbers as a snapshot of that set, not a live frontier ranking. Tasks were synthesized with Claude Sonnet 4.6 from live tool schemas, then filtered by automatic checks and three expert annotators; subtype targeting on a human-labeled failure sample landed at 81% intended-primary (κ = 0.78). The public repo is J1zz/mcp_hallu.

The practical takeaway is blunt: an MCP agent that looks strong on aggregate task completion can still be weak at recognizing when a tool inventory simply cannot do the job — and that weakness shows up as confident false completion rather than an honest stop.