600 experiments on LeWM: the JEPA world model predicts Push-T about as well as the simulator, and loses its points in planning

Haddy Alchaer spent a few months running about 600 experiments on LeWM (LeWorldModel, arXiv:2603.19312), the small JEPA world model from Lucas Maes, Quentin Le Lidec, Damien Scieur, Yann LeCun and Randall Balestriero. His write-up, published on November 13 and on the Hacker News front page today, makes one claim worth taking seriously: on the Push-T robot-pushing benchmark, LeWM's predictions are already about as useful as the simulator's real physics. Nearly everything it loses, it loses in planning and in how the goal is specified.

That is a different story from "JEPA world models need to get better at predicting the future", and the numbers behind it mostly hold up when you check them. Some of the "discoveries" are older than the blog suggests, and nothing has been released that would let anyone reproduce them yet.

What LeWM is, and how it is usually tested

LeWM is a 15M-parameter JEPA (joint-embedding predictive architecture) trained end to end from pixels on a single GPU in a few hours. An encoder turns each frame into an embedding, a predictor rolls embeddings forward given actions, and a Cross-Entropy Method planner searches for the action sequence whose predicted end state lands closest to the embedding of a goal image, replanning MPC-style as it goes.

The paper's appendix spells out the Push-T protocol: a 50-step evaluation budget, with the goal "sampled 25 timesteps in the future" along the same expert demonstration the start state came from. So the published benchmark tests the short-range regime only.

Prediction is not the bottleneck

Alchaer's first experiment swapped LeWM's learned predictor for the simulator itself, keeping the planner and the expert subgoals the same, on 250 tasks with the goal 75 steps away. LeWM succeeded 78% of the time; perfect physics, 77%.

Then he took the subgoals away. Planning all 75 steps at once, as the published setup does, LeWM succeeded on 22% of those tasks. Exact physics in the same setup managed 42%. Giving LeWM a new subgoal every 25 steps, the range where it already reaches the goal 89.7% of the time (90.5% with exact physics), lifted it to 80%. Running the planner four times and keeping the cheapest plan cost four times the compute and added under a point.

The subgoals there came from an expert demonstration, which a deployed robot would not have. His replacement is a graph over about 460,000 LeWM embeddings of the training demonstrations: states link to their successors and to nearby states in other demonstrations, and a graph search picks the subgoals. With free jumps between demonstrations, the search strung together paths the block could not follow (318 of 1,015 plans used six or more jumps) and success was 62%. Charging a cost per jump brought it to 85%, the same as handing the planner expert subgoals.

The pusher in the goal image

The most transferable result has nothing to do with the model. A Push-T goal is an image, and an image has to show the robot's round pusher somewhere. LeWM's cost compares whole embeddings, so the planner also tries to put the pusher where the goal frame happened to show it.

Alchaer isolated this with the exact-physics planner on 50 tasks from a harder test he built (start and goal taken from different demonstrations, block held on target for the last 10 steps). Scoring block and pusher: 9/50. Scoring the block only: 39/50. Thirty tasks flipped from failure to success and none flipped back; on the flipped tasks the planner had put the pusher within half a pixel of its goal position and lost the block on the way.

He says a 30-to-0 split happens by chance "about twice in a billion". That is right: under a fair sign test, two-sided p = 2 × 0.5^30 ≈ 1.9 × 10^-9.

LeWM's embedding has no separate "pusher" slot, so the fix he tested is a workaround: compare each prediction with 16 versions of the goal image that keep the block fixed and move the pusher to positions taken from stored demonstrations, and use the closest. That doubled LeWM's success on the harder test, from 11% to 23%, with no retraining. The general point is one roboticists will recognize: "put the cup on the shelf" is a set of acceptable end states, and a single goal image quietly turns it into one specific arm pose.

We checked the statistics

Two of Alchaer's side remarks are more important for anyone reading Push-T numbers than his headline methods.

The official 50 episodes look easier than the distribution they come from. He reports LeWM at 68% on the official 50-episode set and 44% on 250 fresh episodes "drawn the same way". If the true success rate were 44%, getting 34 or more successes out of 50 would happen about 0.05% of the time (binomial; a two-proportion z-test gives z ≈ 3.1). So this is not just small-sample noise: either the fixed 50 episodes are an unusually easy draw, or something in the setup differs. One inconsistency to flag: he describes that comparison as "goal 50 steps away", while the LeWM paper's own protocol samples the goal 25 steps ahead within a 50-step budget. Without his code we cannot tell which setting he ran.

One seed on 250 episodes resolves only large gains. Changing nothing but the planner's random seed flipped 38 of 250 episodes. At n = 250 the 95% interval on a success rate is about ±6 points. His biggest effects (22% to 80%, 62% to 85%, 9 to 39 of 50) are far outside that. Smaller ones need more care: transformed memory segments raising the harder test from 45% to 56% is plausible but not decisive from a single run, and the 11% to 23% result does not state its sample size.

His other observation about the benchmark also fits independent evidence. In Push-T, every expert demonstration ends with the block in roughly the same place, and for 84% of the standard test's goals the memory graph already contained a nearly identical state. The H-JEPA paper (Wancong Zhang, Basile Terver, Michael Rabbat, Yann LeCun, Randall Balestriero) ran into the same short, convergent episodes from the other side: its two-level hierarchy "matches or exceeds LeWM at every budget" on Push-T, but its three-level model performs poorly because Push-T episodes are too short to cut enough long training clips (it sees only 58% of LeWM's transitions per epoch).

What is new and what is not

Hacker News readers pushed back on the word "discovered", fairly. Planning over a graph built from stored experience is the idea of Search on the Replay Buffer (Benjamin Eysenbach, Ruslan Salakhutdinov, Sergey Levine, 2019), and hierarchical planning through subgoals is an established research area, not a finding. Commenters also pointed at the dependence on the demonstrations: delete the stored states near the goal and success falls from 93% to 32%, and the mirrored and rotated copies that bring it back to 85% work because Push-T's physics happens to be symmetric under shifts, rotations and reflections. That is a property of a 2D table, not of robots in general.

What the write-up adds is a careful accounting on one public model: how much each part of the system is worth. On Push-T, replacing the learned dynamics with perfect physics is worth about 20 points on long-horizon tasks; chunking the plan is worth almost 60; changing what the cost compares is worth 30 tasks out of 50. For a model family whose case is usually argued at the level of architecture, that is a useful corrective. We saw the same pattern this week in robot-arm agents, where harness changes moved GPT-6 Astra's RoboDojo score from 29.7 to 50.3: the thing being benchmarked is the model plus everything wrapped around it.

The limits are plain. This is one author, one 2D environment and one model; the blog links no code, configs or run logs; and none of it has been checked by the LeWM authors. Alchaer says he is now working on a planner that proposes its own subgoals beyond the demonstrations, where the harder test currently tops out at 56%, mostly failing on goals that run against the direction of the data.