JustRUA (arXiv:2611.16009): GPT-6 Astra's weak robot-arm scores were mostly the harness, from 29.7 to 50.3 on RoboDojo

When researchers first put GPT-6 Astra in charge of a two-armed robot on the RoboDojo simulation benchmark, it looked clumsy at anything that needed precision. A new controlled study says most of that clumsiness belonged to the software around the model. "JustRUA: A Controlled Study of GPT-6 Robot-Use Agents" (arXiv:2611.16009), by Zefan Wang, Yufan Dang, Xuanwei Shi, Mingqi Gao, Cheng Yin, SikYuen Tam, Hanyu Liu, Yasheng Wang, Yaxi Lu, Songyang Zhang, Xin Cong and Yuan Yao (Tsinghua, HUST, Peking University, ModelBest and Tencent), keeps the model, the motion planner and the simulator fixed and changes only the harness. The score on their evaluation set goes from 29.7 to 50.3, with no learned policy, no self-improvement loop and no task-specific tuning.

A "robot-use agent" is the robotics version of a computer-use agent. There is no vision-language-action model in the loop: GPT-6 Astra looks at three camera images, decides where each gripper should go next as a world-frame target, a classical planner (CuRobo) works out how the arm gets there, and the model sees what happened. RoboDojo has 42 bimanual tasks in five groups (Generalization, Precision, Long-horizon, Memory, Open), and since the RoboProbe paper published the official "L3" harness, a run of systems has pushed GPT-6 Astra's score up by bolting on depth sensing, affordance models, demonstrations and self-written skill libraries, usually several at once and on different test subsets. JustRUA's point is to find out which of those parts actually earn their place.

What moved the score

The authors walk one path from the official harness, keeping a change only if it helps on a fixed 111-episode evaluation set at a reasonable cost and doesn't draw on the tasks being scored:

The two changes at the top are the least glamorous and the most instructive. The official harness keeps images from only the last two observations, so the model forgets what it saw a few moves ago, such as an object now hidden behind its own arm. And it executes each planned motion to the end before the model looks again; when CuRobo occasionally returns an unstable plan, the arm swings across the table unchecked. Cutting every motion at 20 control steps (about 0.8 seconds) doesn't make the planner better, it just gives the model a chance to notice. The Precision group, where GPT-6 Astra had looked weakest, rises from 12.7 under the official harness to 25.0 with those two changes and 29.1 in the final configuration.

Depth is the other clear win. Depth images cut empty grasps (the gripper closing on nothing) in the 42 reviewed first-round episodes from 41 to 16, and depth predicted from RGB worked almost as well as perfect depth read from the simulator, which the RoboDojo leaderboard doesn't accept for agent entries anyway. The predicted distances were off by tens of centimetres; the ordering of near and far was right, and that seems to be what the model needed. In the last step, 123 of the model's 132 code runs in the first round were geometry: casting rays from pixels through the camera calibration and triangulating points seen by two cameras.

What didn't

The list of things that failed to help is as useful as the list that did. Relative targets (offsets from the current pose instead of world coordinates) scored lower and made the arm hit objects almost three times as often. Named skills like pick, handover and adjust_view changed behaviour more than results: the model moved its cameras more than three times as often and dropped more of what it held. Precomputed grasp affordances from GraspGen, the most elaborate addition, never beat plain depth from the same source.

The pointed finding concerns experience reuse, the core of several leaderboard entries. Demonstrations looked up by task name and a self-written skill library both changed how the model acted (fewer drops, fewer empty grasps), but neither clearly beat the default, both made episodes more expensive, and both draw on the tasks being scored. The authors' conclusion is blunt: such scores "are not comparable with a standard evaluation." Anyone reading the RoboDojo leaderboard should keep that in mind.

What we checked

We recomputed the overall score of all 19 configurations in the paper's main table from its five group scores (the benchmark weights groups equally). Every row matches to within 0.04 points except the official harness, where the group scores average to 29.0, not 29.7. That isn't an error: the table's footnote says the official row's group scores come from its full 2,100-episode run (29.0 overall), while 29.7 is its score on the authors' 111-episode set. It does mean the headline "29.7 to 50.3" compares two numbers on the same evaluation set, while the group-by-group jumps for the official row, including the Precision figure above, compare against the full run.

We also read the released policy in the ZefanW/JustRUA repository (Apache 2.0). policy.py does what the paper says: CONTROL_CAP = 20, motions longer than that are cut to their first 20 controls, a gripper command in the discarded tail is undone rather than quietly applied on the next decision, the model is told how many controls actually ran, and max_decisions is None. The model is gpt-6-astra at medium effort.

Two caveats come from the paper's own numbers. First, noise: each configuration ran once, and the authors' bootstrap puts the standard deviation of an evaluation-set score at about 2.6 points. Steps of under 2 points, such as the gripper readings (+1.8) or the visual demonstration (+1.9 over the default), sit inside that band. The cumulative climb and the depth step are well outside it. Second, the last step bundles two changes. Removing the 100-decision budget matters on its own: in the move-sequence experiment, most of a 4-point gain came from six episodes that had been running out of decisions, and with those six scored as before the gain shrinks to 1.4 points. So some of the final jump from 42.4 to 50.3 is the budget, not the code.

Then there is cost. The final configuration costs $12.53 per first-round episode, two and a half times the $4.94 default, and the authors call that a lower bound because it bills every repeated input token at the cached price; an episode reads several million input tokens and writes a few thousand. As our own rough arithmetic, at that rate a run the size of the official evaluation (2,100 episodes) would cost on the order of $26,000. The scores also stay in simulation, with one model and one benchmark, and the authors are explicit that their evaluation-set numbers are not leaderboard scores.

The broader lesson is one we keep running into. On FrontierCS 2, a harness that quit on refusals cost Gemini 4 Argon seven tasks. JustRUA makes the same point in a physical setting and in more detail: the unit being measured is the model plus its loop, and a thin wrapper that stops each motion early moved GPT-6 Astra's precision score more than most of the clever additions the field has been stacking on top.