sysnote
Analysis

Coding agents keep checking passing SWE-bench patches: what Termination Lag's data shows

Qwen3-Coder-30B-A3B kept working for a median of 16 steps after producing a patch that passed SWE-bench Verified's hidden tests. It never saw those results. Much of the work that followed went into checking the fix and exploring the repository, so the delay is easier to identify after a run than to prevent while it is happening.

Musa Cim, Burak Topcu and Mahmut Taylan Kandemir of Penn State call this termination lag in “Termination Lag: Coding Agents Solve Early but Stop Late”. I recalculated the headline metrics from the released per-task results. They hold up, but the files also qualify two claims: the abstract's first-edit success range leaves out Haiku's higher result, and some runs counted as losing a correct fix ended at a limit.

Measuring a fix the agent cannot see

The authors saved intermediate patches and tested each distinct patch after the run, using the benchmark's official tests. The agent received none of those grades. A step means one model response together with its tool calls; termination lag is the number of steps between the first passing patch and the end of the run.

For Qwen3-Coder on Verified, the first passing patch appeared at a median step of 30 and runs ended at a median step of 48. The median delay was 16 steps. Those figures are calculated separately across runs: subtracting the two median step numbers would give 18, not the median of the individual delays.

The authors evaluated five models across eight model, benchmark and harness combinations. OpenHands was the main harness; Qwen3-Coder also ran under mini-SWE-agent. Verified had 500 tasks, Multilingual 300 and the Pro sample 300. Haiku and GPT-6 Luna each used 100 Verified tasks. Each task was run once in each evaluated setting.

Model, benchmark Solved First correct step Stop step Extra steps Share after the fix
Qwen3-Coder, Verified 43.8% 30 48 16 35%
Qwen3-Coder, Multilingual 22.3% 34 54 18 37%
Laguna-XS.2, Verified 56.0% 38 95 57 60%
Laguna-XS.2, Multilingual 48.0% 50 108 55.5 54%
Claude Haiku 5.5, Verified (100) 78.0% 9 19 10 54%
GPT-6 Luna, Verified (100) 48.0% 9 15.5 5.5 36%
Seed-OSS-36B, Pro (300) 7.7% 26 45 14 35%
Qwen3-Coder, mini-SWE-agent, Verified 32.4% 22 37 12 37%

The step columns and final percentage are medians over runs whose final patch passed. “Solved” is the share of all tasks in that setting. The extra-step share is calculated within each run before taking the median. The repository's reproduction script specifies these calculations.

Stopping each successful run at its first passing patch would remove a median of 35–60% of its steps across these settings. That is the paper's retrospective oracle: it has access to the hidden grader. The experiment does not supply an equivalent stopping rule an agent could use during a task.

Most later work checks the patch

In the paper's Table 3, verification accounts for 48–63% of post-fix steps: running repository tests, writing and running reproduction scripts, and building the project. Exploration takes another 15–29%. Further source edits account for 2–6%, except for Seed-OSS on Pro at 19%. A benchmark-passing patch can therefore be followed by substantial work without much further editing.

Passing the hidden tests also does not establish that every later check was pointless. Those tests define success for this experiment; an agent still has to decide what evidence available to it is sufficient.

Across the eight settings, I counted 1,074 runs that reached a passing patch. Of those, 53 ended with a failing final result. Temporary regressions were also common in some settings: the Laguna Multilingual action file records a passing patch becoming failing and then recovering in 48 of its 143 analysed successful runs.

Haiku exceeds the abstract's first-edit range

The abstract puts first-edit success at 38–82%, and the introduction says that range holds across all settings. Table 2 supplies results for Qwen3-Coder, Laguna-XS.2, GPT-6 Luna and Seed-OSS, but leaves out Claude Haiku 5.5. Haiku is also absent from Table 3, although it appears elsewhere in the paper.

In results/first_patch.json, Haiku's first source edit passed in 71 of the 79 runs that ever reached a passing patch: about 90%, above the abstract's upper bound. The missing mini-SWE-agent setting comes out at 81 of 175, or 46%, within the stated range.

Haiku's files use different populations for different questions. There were 79 runs with a passing patch at some point and 78 passing final results, out of 100 tasks. Its post-fix action file covers 75 successful runs. All 75 kept a passing patch throughout the recorded interval, and verification accounted for 66% of their post-fix actions. That subset should not be treated as all 79 runs.

Haiku still had a median delay of 10 steps, or 54% of a successful run. A high first-edit success rate did not make the agent stop as soon as its patch passed the hidden tests.

A failing final result does not always show an undone fix

The paper describes the 1–8% of runs ending unsuccessfully after a passing patch as agents undoing an earlier fix. The released records distinguish some cases that this description groups together.

In 11 of the 53 runs, finished is false. The data format defines that as a step cap, context limit or time limit ending the run. In five of those 11, the last recorded patch still passed, despite a failing final result.

The submitted patch is graded separately from the intermediate snapshots. Those five records show a discrepancy that needs explaining before they can be counted as agents breaking their own fixes. The fields alone do not establish how the passing snapshot became a failing submission. The other 42 runs ended voluntarily with a failing final result.

Step counts do not give a cost saving

The paper reports median context growth between the first passing patch and the final step of 30% and 28% for Qwen3-Coder, and 78% and 55% for Laguna-XS.2, on Verified and Multilingual respectively. Later requests can therefore involve more input, but their cost also depends on caching and output length. Removing 35% of steps does not establish a 35% reduction in billed tokens, money or elapsed time.

The remaining verification limit concerns the action labels. The result files let a reader recalculate when patches passed, when runs ended and how those grades changed. The recording code, stepproxy.py, takes snapshots through a temporary Git index, leaving the agent's working copy alone. That explains how patches were captured, but the grades cannot establish whether a step ran tests, inspected code or did something else.

The README links complete messages and tool calls to musacim/termination-lag-trajectories. I could retrieve the released result files for this check, but not those trajectories. Checking the authors' action classifications against the actual commands requires that missing record.

Comments

No comments yet. Be the first.

Write a comment

Comments are published after review.