GVS5H's Qwen3.8-27B vs Claude Fable 5 claim, checked twice: the grading holds, the 2-point lead doesn't

"Five Qwen3.8-27B Models Match Claude Fable 5 on LiveCodeBench Hard" is the headline of the GVS5H repository, and on October 25 it took the project to the top of Hacker News. In the week since, two outside groups have checked it from different ends. One regraded every stored solution in the authors' repository; the other rebuilt the method in a separate tool and ran it again. Both say the authors' numbers hold. Neither says Qwen3.8-27B beats Fable 5, and the regrade shows how much of the "match" rests on choices that are easy to miss in the abstract.

The claim

The paper, GVS5H: Zero-Shot Self-Orchestration with Ledger-Based Control for Improved LLM Coding Performance by Victor Gao, Juhyun Lee and colleagues (the repository is published under Persis Capital's research lab), describes a training-free harness. Fresh instances of one model play manager and workers and coordinate only through files on disk: a plan, notes and the current solution. On LCB-100, the 100 most recent hard LiveCodeBench problems, the paper reports that Qwen3.8-27B rises from 66.8% with one call to 92.4% with the manager, against 90.4% for Claude Fable 5 called once. Orchestrated Qwen3.8 Flash Next reaches 93.0% at 9% of Fable 5's cost. The abstract also says gains "are not universal: some models are unchanged or worse".

The regrade: the grading holds, the lead doesn't

On October 31, CyberNative AI, which describes itself as an AI-run company, opened an issue on the GVS5H repository with the results of an independent regrade. It wrote its own judge from the problem statements and LiveCodeBench's comparison rules, without reading the authors' patched grader, and ran it on all 2,500 stored solutions of five arms (five passes over 100 problems). It did not rerun any model.

What it found, in order of how much it matters:

• The authors' corrections are real. The paper's scores come after two fixes to LiveCodeBench's evaluator, described in its Appendix B. With the stock evaluator, the 27B manager arm scores 84.4% and Fable 5 scores 87.4%, so before the corrections Fable 5 is ahead. The fixes moved 40 of the 27B manager's solutions from fail to pass; 39 of them pass the outside judge at 6 seconds and all 40 at 18 seconds. In 2,500 rows the only disagreements with the paper are timeouts.
• A two-point lead is not a lead. The 27B manager arm minus Fable 5 is +2.0 points, with a 95% interval of −1.0 to +5.4. The regrade notes that the paper's own Table 1 says the same thing (p = 0.88). "Match" is a fair word; the abstract's "or slightly exceed it" is not something these 100 problems can show.
• Fable 5's refusals are worth 1.2 points. Fable 5 refused 6 of its 500 attempts, 5 of them on two LeetCode problems, and each refusal counts as a failure. Most of the 27B arm's edge comes from the 27 LeetCode problems (97.8% against 91.9%); on the 73 AtCoder problems the two are at 90.4% and 89.9%. The paper discloses the refusals.
• The time limit decides the last point. The authors' rescoring script defaults to 10 seconds per test; LiveCodeBench's default is 6, and the paper states neither. On two problems with 3.4 MB and 4.4 MB inputs, the 27B manager's solutions run close to the limit. At 6 seconds on the regrader's machine the arm scores 91.0–91.4% and Fable 5 doesn't move. At 18 seconds everything matches the paper, but the 27B's passing solutions there took up to 15.9 seconds, Fable 5's at most 3.9.

The regrade also checked two common objections against the authors' own verdicts. The gain is not just the manager running public tests: the loop runs them only on AtCoder problems, and the 27B gains about as much on LeetCode, where it doesn't. Nor is it plain resampling: a perfect pick from five independent single calls of the 27B would reach 90.0%, below the manager's 92.4%.

The rebuild: parity on ten problems

The second check comes from guregodevo, the developer of the agent tool Memdoor, who rebuilt the method as a Memdoor workflow called team-solve: eight YAML files, a planner, two ideators, a solver and a manager, with every request checked byte for byte against the authors' code. It was posted to Hacker News this weekend under a louder title ("beats Fable 5") than its own README uses.

The README is more careful. On the first ten problems of the authors' list, five passes each, the workflow solved 9.8 on average, the authors' harness 9.6 and Fable 5's single call 8.0. "On a sample this small, 9.8 and 9.6 are the same number," it says; the claim is parity with the authors' harness, not a gain. Fable 5's column comes from the authors' files and was not run in this harness, and the model ran on OpenRouter hosts rather than a local machine. A run took a median of 40 minutes and $0.39; all 50 runs cost $35.79.

The working notes are worth reading for the failures. The one problem lost was lost to the method itself: the manager invented a counterexample with a wrong expected answer, and the team rewrote a correct program into one that passed the samples but not the hidden tests. Twice more, a program that passed every hidden test was overwritten in a later round and had to be won back before the loop ended. A manager that can't be checked can talk a team out of the right answer.

What nobody has tested

Contamination. The paper says taking the most recent hard problems "limits training-data contamination", but the 100 problems are dated December 7, 2024 to April 6, 2025, while the Qwen3.8-27B repository on Hugging Face was created on August 5, 2026. Neither the paper nor the model card gives a training cutoff. The regrade asks the authors to state one, or to run the models on problems published after their release; as of November 1 the issue has no reply.

The comparison is also lopsided by design. Fable 5 gets one call; the 27B gets a manager, workers, several rounds and, on AtCoder, the public samples. That doesn't make the result wrong, but it measures a harness against a single call, not one model against another. We have seen the same lesson in FrontierCS 2, where how a harness handled refusals moved a score more than the model did. Here the refusals sit on the other side of the table.

What survives is still interesting: on a hard, if old, competitive-programming set, a 27B open-weight model in a simple file-based loop lands within noise of a frontier model's single call, and two outside checks found the bookkeeping sound. What doesn't survive is "beats".