MCP vs CLI for agents (arXiv:2610.34298): no winner across 18 settings, and the study's "MCP" is a JSON format in the prompt

Is it better to give an agent its tools through MCP or as command-line commands? A controlled study posted to arXiv on October 25, "Matched Capabilities, Different Behaviors: A Controlled Study of MCP and CLI Interfaces for Language Agents" (arXiv:2610.34298) by Zhenning Shi, Liang Pang and Yongqi Li (The Hong Kong Polytechnic University and the Institute of Computing Technology, CAS), gives the question its most careful treatment so far. Its answer is that neither wins. Its most quoted number, that paired MCP and CLI runs diverge in 94.3% of 2,004 episodes, needs one qualification that the abstract doesn't spell out, and the authors' code at ModalityDance/MICA makes it concrete: the "MCP" in this study is not the Model Context Protocol running, but a JSON format the model writes in plain text.

What the study does

The authors built MICA (Matched Interface Comparison and Analysis), which takes one "semantic action contract" per environment (which operations exist right now, with which typed arguments) and renders it two ways. The MCP version shows each operation as a named tool with a structured argument object; the CLI version shows it as a command with flags. Whatever the model answers is decoded back into the same normalized action and run through the same validator, compiler and environment. Only presentation, output syntax and the wording of feedback differ.

They ran six models, Qwen3-4B, Qwen3-8B, Qwen3-14B, Qwen3-32B, DeepSeek-v4-Flash and Gemini-2.5-Flash, on ALFWorld (134 tasks), WebShop and ScienceWorld (100 each), at temperature zero. That makes 2,004 MCP–CLI pairs.

No winner, and a small typical gap

Table 1 of the paper has the scores. We counted them: across the 18 model–environment combinations, MCP scores higher in 8 and CLI in 10. The median difference is 3.9 points.

The headline-sized gaps come from one model. Qwen3-14B on ALFWorld scores 20.1 with MCP and 56.7 with CLI, a 36.6-point gap, and then flips on WebShop, 64.4 against 59.5, while using 35k fewer tokens per task with MCP. Averaged over the three environments, the absolute MCP–CLI gap is 15.3 points for Qwen3-14B, but 2.2 for DeepSeek-v4-Flash, 3.1 for Qwen3-32B and 4.8 for Gemini-2.5-Flash. DeepSeek-v4-Flash scores 99.3 and 95.5 on ALFWorld either way.

There is also a third column the abstract doesn't mention: "Native", each benchmark's own action strings such as go to desk 1, kept as a calibration reference. It uses the fewest tokens of the three in 13 of the 18 combinations and scores best or tied in 7. Qwen3-14B on ALFWorld gets 62.7 with it. For several of these models, both structured interfaces cost more than the plain text the benchmark already had.

What "MCP" means here

The study's "MCP" is an output rule in the prompt:

rule = (
    "Return exactly one JSON object inside one <tool_call>...</tool_call> container, "
    "with no text before or after it. The object has exactly name and arguments: copy the "
    "operation's exact tool_name into name, and put its argument keys and selected values "
    "INSIDE the arguments object. Use JSON double-quoted keys and string values."
)

The CLI counterpart asks for one command inside <command>...</command>, with multiword values in POSIX single quotes. Nothing is executed in a shell, and the project's pyproject.toml has no MCP SDK among its dependencies. There's no JSON-RPC, no tools/list, no MCP server, and no provider function-calling API: the model client sends every model, the API-served DeepSeek-v4-Flash and Gemini-2.5-Flash included, a plain chat.completions request with messages and no tools parameter, and reads the answer back as text. The shared system prompt calls the two formats "controlled CLI-style" and "controlled MCP-style syntax", which is accurate. "MCP" in the paper's title is shorthand for it.

That design is the point of the paper, not a flaw: to isolate representation, both arms must share one runtime. But it means the study answers a narrower question than the one people argue about. The practical MCP-vs-CLI debate is about real systems: tool schemas loaded into context, native tool-call training, CLIs the model already knows from pretraining like git or gh, and shells that compose commands with pipes. None of those differ between the two arms here. The paper itself cites two system-level comparisons for that side: "Terminal agents suffice for enterprise automation" (arXiv:2604.00073) and "The scaffolding matters more than the interface" (arXiv:2608.08654).

What the 94.3% measures

The 94.3% (1,889 of 2,004 pairs) counts episodes whose "interaction signature" (the decoded action, its validation, the execution outcome, the feedback and the recovery record) differs at some step. Runs are deterministic at temperature zero, so a single different first action is enough, and the paper finds divergence starts near the beginning of interaction and that semantic choices, not just syntax errors, account for most of it. That is a genuine finding about how path-dependent agent loops are. It is not a claim that 94% of tasks end differently; final scores, as above, are usually a few points apart.

The failures are mundane in a way agent builders will recognize. One WebShop example in the appendix is a product option written 18''x18'', which has to survive POSIX quoting inside a --target flag; the JSON arm has its own decoding failures. In matched-state probes, changing only the output syntax shifts realization failures by up to 84.4 points for Qwen3-32B on ALFWorld.

DART, the fix

The authors' remedy, DART (Decoupled Action Realization and Targeted Repair), first gets the model to commit to a semantic action and then, if the formatted call fails, repairs only the formatting instead of letting the model pick a new action. On Gemini-2.5-Flash, the only model it was tested on, it adds 0.8 to 9.0 points in all six benchmark–interface settings and cuts MCP-style token use by 4.3k to 27.1k per task. Under CLI it increases tokens in two of three environments. It's a sensible pattern for any agent harness: when a tool call fails to parse, ask for the same call again in valid form rather than throwing the decision away.

What to take from it

For small and mid-size models, the output syntax you ask for can move results by as much as 36 points, in either direction, depending on the environment. For DeepSeek-v4-Flash and Gemini-2.5-Flash, the effect is a few points. The models tested are a generation or more behind today's frontier, the episodes are temperature-zero runs, and ScienceWorld histories are cut to eight events. If you're choosing between an MCP server and a CLI for a production agent, this paper says to test with your own model and harness. It doesn't say which one to pick.