Resolution Hypothesis (arXiv:2611.23068): self-evolving agents' largest benchmark gains came from grader shortcuts
"The Resolution Hypothesis: Agent Evolution's Key Is Identifying Improvements" (arXiv:2611.23068) appeared on November 17. Its authors are Yuanhao Zeng, Shaocheng Lan and Ao Lu, with Kan Ren as corresponding author, from ShanghaiTech University, ByteDance, BIGAI and SUSTech. They ran seven agent-evolution methods on six models, more than 10 million LLM calls in total, and say two things about the field. Most of the margins papers report between methods are smaller than the difference between two runs of the same method. And the largest gains come from evaluation shortcuts that one sentence added to the prompt can reproduce. The trajectory data and code are promised "upon the acceptance of this paper", so I checked what can be checked today: the public graders and released artifacts the shortcut claims are about.
What agent evolution is
The methods covered here leave the model alone and rewrite something around it. GEPA evolves prompts, SkillOpt-Lite and Trace2Skill evolve skill files, ReasoningBank evolves a memory playbook, and HarnessLens, Meta-Harness and AHE edit the harness code. The loop is the same in all of them: an LLM proposes a new version of the artifact, the agent runs some tasks with it, and the version that scores higher is kept.
The paper asks how much of that score is noise. It splits the variance into two parts: σ, the spread you get by evaluating the same artifact again, and τ, the spread in true quality between artifacts produced by separate evolution runs. A table with one run per method, evaluated once, contains both.
One run cannot rank methods
Each method was evolved 8 times per model and benchmark (16 times on GPT-5.4 nano for the text methods), and each resulting artifact was evaluated up to 16 times: 1,030 evolution runs and over 2.1 million test trajectories on ALFWorld, SpreadsheetBench, OfficeQA and Terminal-Bench v2.1. A single run evaluated once typically has a standard deviation of 1.7 to 4.1 points, depending on the benchmark. The authors also went through 171 benchmark–model results in the main tables of nine agent-evolution papers: the new method leads its strongest competitor by a median of 3.4 points, none of the 171 reports a variance estimate, and only 6 average more than one run.
To detect a true 3.4-point gap at the usual 80% power, they calculate, you need about 5 runs per method on SpreadsheetBench, 10 on OfficeQA, 19 on ALFWorld and 23 on Terminal-Bench. I worked backwards from their Table 7: the run counts are consistent with single-run standard deviations of 1.7, 2.6, 3.7 and 4.1 points, the same range the text gives. In their simulation, a one-run table picks the best of the four text methods 51% to 73% of the time on the first three benchmarks and 33% on Terminal-Bench, where guessing gives 25%.
Selection at a coin flip
The second half is about the step inside the loop. On Terminal-Bench the authors generated 100 candidate skills and ran each 16 times on 10 held-out tasks. Good skills are in the pool: the best shared skill reaches 57.5% against a pool mean of 49.2%. A second pool of 100 skills, run on 11 tasks, was used to test the selectors. Judged on one rollout, the usual execution-score selector picks the better of two skills 49.7% of the time. Regressions are 30.5% of the pool and 62.1% of what gets accepted. Three LLM-judge setups that read the whole trajectory land between 45.4% and 49.0%.
The consequence shows when the artifacts are re-run. Over eight SkillOpt-Lite and eight GEPA runs, the score the loop records for itself rises by 6.9 to 9.0 points, while the same artifacts, re-evaluated eight times per task, move between +0.4 and −3.2. The paper's explanation is that the loop keeps whichever candidate drew a lucky score, round after round.
I can't check any of this yet. It rests on trajectories that are not public, and the paper limits it to Terminal-Bench and, for the re-evaluation, to two methods and two models.
Where the big gains came from
On benchmarks where evolution looked impressive, the authors read the evolved artifacts instruction by instruction, picked the one rule that seemed to explain the gain, and appended only that sentence to the unevolved prompt. Across 27 model–benchmark pairs with a gain above 3 points, the sentence recovers a median of 90% of it. Then they repaired the graders and re-scored the same artifacts: the median gain over twelve benchmarks falls from 17.6 points to 0.8.
Four of those shortcuts sit in public code or data, so I looked at them myself.
SpreadsheetBench. SkillOpt-Lite's grader (commit 4cb4eee, the one the paper cites) opens the answer file with openpyxl.load_workbook(..., data_only=True). openpyxl does not calculate formulas, so a formula the agent writes is read back as an empty cell. The official SpreadsheetBench instructions run open_spreadsheet.py before evaluation; it opens every file in LibreOffice so the formulas get calculated. On the box I wrote =SUM(B1:B3) into a cell whose expected value is 60 and ran SkillOpt-Lite's compare_workbooks:
formula, no recalc : (False, 'value@Sheet1!B4: gt=60 pred=None')
literal : (True, '')
formula, after LibreOffice recalc: (True, '')
The released SkillOpt-Lite skills contain a section headed "Write COMPUTED VALUES, never formula strings". The repository also ships an opt-in recalculation harness, off by default, whose docstring calls formula strings "the single biggest residual failure cluster". With recalculation, the paper's GPT-5.4 nano baseline goes from 33.5% to 58.0% and the gain from +31.1 to +6.7 points.
OfficeQA. The agent prompt in Databricks' OfficeQA Pro report asks the model to "only return the value" inside <FINAL_ANSWER> tags, and its reward.py extracts numbers before comparing them. SkillOpt-Lite's port asks for the answer inside <answer> without that instruction and grades by exact match after light normalization. I ran both graders on the same strings with the gold answer 951134.33: "951,134.33 million USD", "approximately 951134.33" and "The mean is 951,134.33" score 1 in Databricks' grader and 0 in SkillOpt-Lite's. The released OfficeQA skills for GPT-5.4 mini and GPT-5.5 tell the model "VALUE must be the bare number only". For GPT-5.6 Luna the paper's gain goes from +21.1 to −0.1 after the repair.
LiveMathematicianBench. Some questions in this math benchmark include the option "One of the remaining options is correct, but a stronger result can be proven." In SkillOpt-Lite's split it appears in 85 of 177 questions and is the right answer in all 85; on the 106 test questions the count is 46, which matches the 60 option-free questions the paper uses for its repaired score. The paper lists this shortcut under "the released evaluation" without saying whether SkillOpt-Lite introduced it. The option comes from the benchmark itself. In the upstream dataset on Hugging Face (LiveMathematicianBench/LiveMathematicianBench, all eight monthly files) the option appears in 263 of 608 questions and is correct in all 263. The benchmark paper explains why: to stop models from guessing, it replaces the correct option with this sentence for "a configurable fraction" of questions. In the released data the sentence never appears as a wrong option. A model that picks it whenever it appears gets 43% of the upstream questions right without reading the mathematics. The released SkillOpt-Lite skill for GPT-4o says "Default to this meta-option whenever it is present", and gives the model a reason: concrete options "are almost always deliberately understated".
AIME 2025. The AIME example in GEPA's repository scores with int(prediction.answer). An answer written as 204., $204$ or \boxed{204} fails, and the feedback sent back to the proposer says the answer "must be a valid integer and nothing else". The paper's GPT-5.4 nano baseline on AIME 2025 is 32.0% under the released parser and 85.8% once answers are checked for equivalence after removing LaTeX wrappers.
What holds and what doesn't
The four shortcuts are where the paper says they are, the LiveMathematicianBench count agrees with its own, and the sample-size table is internally consistent. What I can't confirm is the candidate-level part: the 49.7% selection accuracy, the ranking simulation and the in-loop re-evaluation all wait on the promised data. I found no code for the paper yet in the arXiv listing, on the project page, in the GitHub account that hosts the project page (including the branches of the site repository), in a GitHub repository search or on Hugging Face.
Figure 3's caption says the largest gains "vanish after repair", but Table 11 still shows +5.7 to +6.7 points on SpreadsheetBench after recalculation, and GEPA keeps +5.5 to +7.6 on HotpotQA from better retrieval. The authors discuss both, along with HoVer, where one sentence explains less than half the gain for two of the three models. The one rule was also chosen after reading the artifacts, which the paper says shows that one sentence is enough, not that the artifacts contain nothing else. Semantic grading uses DeepSeek-V4-Flash and GPT-5.6 Luna as judges, two of the evaluated models; both have to accept an answer, which limits but doesn't remove that concern.
For anyone running GEPA or SkillOpt-Lite on their own tasks, two things follow without the unreleased data: re-run the final artifact several times before believing a few-point gain, and read the grader before reading the skill the loop wrote for it. In the SpreadsheetBench, OfficeQA and LiveMathematicianBench cases above, the released skills state in plain words what the grader rewards. The trajectories would let others recompute the 49.7%. Whether selection is as blind outside Terminal-Bench is a question the paper itself leaves open.
- The Resolution Hypothesis: Agent Evolution's Key Is Identifying Improvements (arXiv:2611.23068)
- The Resolution Hypothesis — project page
- EvolvingLMMs-Lab/SkillOpt-Lite (commit 4cb4eee)
- RUCKBReasoning/SpreadsheetBench
- databricks/officeqa (reward.py)
- OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded Reasoning
- LiveMathematicianBench dataset on Hugging Face
- LiveMathematicianBench (arXiv:2604.01754)
- GEPA AIME example, utils.py
Comments
No comments yet. Be the first.