A web agent trained under one harness drops from 85% to 19% success when its input fields are reversed

Agent harnesses, the prompts, tools and control loops that sit between a model and its task, are usually compared by keeping the model fixed and swapping the harness around it. A paper from three researchers at the Hong Kong University of Science and Technology (Guangzhou) argues that this comparison is biased against new harnesses whenever the model has been post-trained under an old one. Its most striking example: a small web agent loses most of its skill when the fields of its input are listed in reverse order, even though nothing about the task has changed.

A policy that has learned its harness

The theory is a piece of reinforcement-learning bookkeeping. Under KL-regularized training, the standard setup for RL post-training, the best possible policy prefers each action over another in proportion to how much the harness rewards it. A model trained toward that optimum therefore ends up carrying the harness's preferences in its weights, but only as read through that harness's prompt format. The authors then split the return of any deployed agent into two parts. The ceiling is the best result the harness allows. The mismatch is what the model loses by acting differently from the optimal policy. Training changes only the mismatch. Swapping the harness changes both.

That is why a fixed-model comparison can mislead. A new harness may offer a higher ceiling, but a model trained on the old one arrives with the wrong habits, so its mismatch jumps and the new harness looks worse.

The evidence, and how small it is

The main experiments use Qwen3.5-2B, post-trained with LoRA on one RTX 4090 under six harness variants, on three MiniWoB++ tasks: ticking checkboxes, ticking them by synonym, and clicking numbered tiles in order. Each task is compiled into a finite graph with one token per action, so the optimal policy and the model's full action distribution can be computed exactly. That precision is what makes the experiments clean. It is also why they say little directly about frontier coding agents.

Within that setting the results are sharp. With the order of the observation fields reversed, and the actions, rewards and optimal decisions all unchanged, the trained model changes its most likely action at 52% of states and task success falls from 85.0% to 19.2%. Renaming the action codes costs 17 to 28 points. Showing the menu as a list costs nothing.

The second experiment adds batched clicks, two clicks in one decision. Moved over without retraining, the model's success drops from 85.1% to 72.9%, so a frozen comparison would reject the new harness. After 1,024 further training updates under each harness from the same checkpoint, the model adapted to batched clicks reaches 88.2%, against 83.4% for the one that kept training on the original. Across all six harnesses, nine swaps were rejected by the frozen comparison while leaving room to improve. The seven that raised the ceiling all came out ahead after adaptation on the paper's main measure, regularized return. In task success the gains ran from +3.1 to +12.6 points, with one exception: a swap to a harness that only adds instructions gained return but lost half a point of success.

The coding-agent test rests on one decimal place

The paper's only contact with real coding agents is a frozen Qwen3.5-9B run through five open harnesses on Terminal-Bench Lite's 39 tasks, three runs each. The harness alone moves accuracy from 47.9% (DeepAgents) to 64.1% (Terminus-2), which is itself a useful data point in our continuing coverage of how much a harness decides a score.

The authors then estimate ceiling and mismatch using GPT-5.6 Luna as a stand-in for the optimal policy, from trajectories on three of the tasks. Ceiling minus mismatch orders all five harnesses exactly as accuracy does, while each term alone does not. The top two places, though, are separated by 0.297 to 0.296 on that estimate. The accuracies they are ranked against, 64.1% ± 5.1 for Terminus-2 and 59.8% ± 7.8 for mini-SWE-agent, overlap within their run-to-run spread. A perfect rank correlation over five items, built on a three-task probe, is a suggestive result, not a measurement tool yet.

One finding from that test does read as practical. DeepAgents loses most because of how many calls it makes, not how bad each one is: its per-decision mismatch is close to Terminus-2's, but the model makes nearly three times as many calls, 14.2 per trajectory against 4.9, and the losses add up.

What to take from it

The code is public, created on October 27. The recommendations are cheap to follow: report a harness's ceiling alongside the frozen score, test several checkpoints per harness instead of one, and check whether pure presentation changes move the model's choices. For anyone reading agent leaderboards, the paper gives a formal reason for a familiar suspicion. When a lab's model scores best in that lab's own harness, part of the score may be the model having learned that harness's habits during training. What the paper doesn't show is how large that effect is for frontier models with harnesses far richer than a menu of single-token actions.