Identically trained LLMs break on different jailbreaks: a new pretraining seed matters almost as much as new data, and the released labels back it up

Run the same pretraining recipe twice with a different random seed, give both models exactly the same safety training, and they will not fall for the same jailbreaks. That is the finding of Identically Trained Language Models Break on Different Jailbreaks, a preprint by Chi-Wei Chang, Mu-Tien Kuo, Kuan-Wei Lee and Meng-Hsun Tsai (Cerulean Labs, National Taiwan University and National Yang Ming Chiao Tung University). It sounds like a footnote for people who train 1B models. It is really a statement about what a jailbreak evaluation of a single checkpoint can and cannot tell you.

We pulled the authors' released per-prompt labels and recomputed the central comparison ourselves; the numbers hold up (details below).

The setup

The study leans on Ai2's DataDecide suite, which already pretrains a 1B model on each of 25 data recipes with three different pretraining seeds (same 100B tokens, same code; the seed sets initialization and data order). The authors took every one of those 75 base models and ran the same safety post-training on each, the OLMo 2 1B recipe: Tülu 3 SFT, which carries WildGuardMix, WildJailbreak and CoCoNot safety data, followed by DPO. Each base model got two post-training seeds, which only shuffle the order of training examples. That is 150 models.

Every model then answered 3,212 adversarial prompts (harmful requests wrapped in a jailbreak, drawn from HarmBench, SORRY-Bench, StrongREJECT, WildJailbreak and others) 20 times each, and WildGuard labelled whether it refused. For each model, that gives a "vulnerability profile": the non-refusal rate on every prompt. The question is how similar two profiles are, depending on what the two models share.

What they found

The authors borrow a twin-study vocabulary. Two models built on the same base model, differing only in post-training seed, are "MZ twins". Two built on different pretraining runs of the same data are a "same-recipe pair". Two built on different data are a "cross-recipe pair". Correlations below are corrected for the sampling noise of 20 draws per prompt.

The interesting gap is the first one. Swapping the post-training seed barely moves which prompts break a model. Swapping only the pretraining seed drops the agreement by 0.173, and then changing the entire pretraining corpus on top of that costs a further 0.032. By the authors' arithmetic, not sharing a pretraining run moves the profile 5.4 times as much as not sharing a data recipe. The ordering holds in all 25 recipes, and at 150M and 300M parameters too.

The practical versions of that number are blunter:

• Attack rankings don't travel. The most effective of HarmBench's 91 human jailbreak templates on one checkpoint stays the most effective after a new post-training seed in 67% of pairs, but after a new pretraining seed in only 27%. A 10-to-20-point lead of one template over another is reversed by an MZ twin 2.2% of the time and by a same-recipe model 25.3% of the time.
• Reliable breaks aren't reliable elsewhere. A prompt that breaks a model at least nine times in ten does the same to its MZ twin 75.0% of the time, to another run of the same data 60.9%, and to a model on different data 58.7%.
• The profile is a fingerprint. Matching a model's profile against all 75 runs finds its own pretraining run 75 times out of 75, even after a different post-training seed. A random 50 prompts are enough 98% of the time. The same method identifies the data recipe of an unseen run only 48% of the time.

The data recipe is not irrelevant: it explains 49% of the variance in how often a model breaks overall, against 32% for the specific base model. It just says little about which prompts do it. And even the "how often" question is shaky with one model per recipe: comparing two recipes through one model each gets their order wrong in 23.4% of comparisons.

It isn't the seed, it's the run

One natural reading is that the seed itself, the initial weights or the data order, bakes in the vulnerabilities. The authors tested that and it is wrong. They pretrained one DataDecide run twice more from the identical seed on their own H200s, starting from bit-identical step-0 weights. GPU nondeterminism splits the runs at step 1, and by the end the reruns sit 94% as far from the original in weight space as runs with different seeds. After the same safety training, the reruns correlate with the original at 0.839 to 0.851: barely above an ordinary same-recipe pair (0.814), nowhere near MZ twins. Partial reruns sharing only the initialization or only the data order added no detectable agreement at all.

So the vulnerability profile belongs to the particular trajectory a run happened to take, which nobody controls even with every seed fixed.

Our check of the released labels

The authors released code under Apache 2.0 on GitHub, the post-trained models as branches of 27 Hugging Face repositories, and the per-response judge labels as the castor-labels dataset (208 models, prompts referenced by benchmark ID only, no generations). We downloaded the labels for 12 of the 1B models of two recipes, DCLM-Baseline and C4 (three base models times two post-training seeds each), and recomputed the profiles on HarmBench's 1,212 human-jailbreak prompts, applying the same corrections the paper describes: coherent samples only, Spearman's attenuation correction with odd/even split-half reliability.

Our ancestry gap on this subset, 0.211, is in line with the 0.210 the paper reports for these human-jailbreak prompts, and the carry-over rates track the paper's 75.0/60.9/58.7% closely even though we used two recipes instead of 25. The pattern is in the data, not an artefact of the paper's aggregation.

The same files also show the judge caveat the authors are upfront about. WildGuard puts these models' non-refusal rate on the human-jailbreak prompts at 44–63%. The authors' own audit says WildGuard misses refusals on jailbreak prompts: it puts mean vulnerability at 0.528, which their audit adjusts to 0.278. That inflates absolute rates, but not obviously the comparison: on 1,200 responses from four models hand-labelled blind by two authors, the MZ-versus-same-recipe gap was still 0.156 (interval 0.036 to 0.294), against 0.203 from the judge on the same responses.

What it means, and what it doesn't

Labs and red teams publish jailbreak results for one checkpoint and talk about them as properties of a model family: "the model is vulnerable to roleplay framing", "our data mix made it more robust". This paper says that, at least at 1B, a large share of which jailbreaks work is an accident of one training trajectory. A red-team finding transfers well across fine-tuning runs of the same base (the 67% and 75% numbers) and badly across pretraining runs, so a model retrained from scratch on the same data may need red-teaming again. Hughes et al.'s Best-of-N work found that prompts rank similarly in difficulty across models; this result does not contradict it, since templates and requests still carry shared difficulty, but once the authors strip those out the gap between MZ twins and same-recipe pairs grows from 0.210 to 0.525.

The limits are real. Everything is DataDecide models at 1B and below, with one post-training recipe (SFT plus DPO, no RL stage); a second safety dataset reproduced the gap, but the training algorithm was never changed. Whether frontier-scale models trained with far heavier safety pipelines are equally idiosyncratic is untested. The paper is honest about why: it took 11,396.6 H200-hours, and a single 7B pretraining run costs on the order of 15,000 A100-hours by an earlier estimate the authors cite. Nobody is training three copies of a frontier model to find out.

What's left is a methodological point that is cheap to act on even without new runs: an attack ranking or a data-recipe comparison read off one checkpoint is evidence about that checkpoint. If you have several runs, evaluate several runs.