Baidu releases ERNIE 5.5 Flash Base weights: 280B MoE, 14B active, under a $50M-revenue license

Baidu has released ERNIE-5.5-Flash-Base, the raw pre-trained checkpoint of its upcoming ERNIE 5.5 Flash: a sparse mixture-of-experts model with 280B total parameters and 14B active per token, trained from scratch on 33.5T tokens. The weights stop at the end of pre-training, with no mid-training or post-training, and are on Hugging Face as baidu/ERNIE-5.5-Flash-Base (FP8) and a BF16 variant. Baidu says the post-trained ERNIE 5.5 Flash "will follow", without a date.

Releasing a base model is the less common choice, and Baidu pitches it as a clean starting point for domain adaptation, instruction tuning and RL research: one checkpoint against which training methods can be compared.

The architecture

The model has 43 layers, and most of them barely look at the past in detail. Only one layer in seven, six in total, keeps token-level access to the whole context, using Multi-head Latent Attention with sparse attention, DSA (a lightweight indexer picks the top-k past positions). The other 37 layers use what Baidu calls Heavily Compressed Attention, squeezing every 128 tokens into one cache entry and adding a short uncompressed window of recent tokens. Training context reached 128K, and Baidu says fine-tuning and inference support 1M tokens.

The rest of the list:

• MoE: 512 routed experts (top-10) plus one shared expert. Experts run in a 2048-dimensional latent space instead of the full 4096, which Baidu says roughly halves dispatch traffic and expert FLOPs. Selection uses several independent sigmoid routers instead of one, which Baidu says makes routing scores track expert usefulness better.
• Residuals: manifold-constrained hyper-connections with four parallel residual streams instead of a standard residual.
• Output head: "MultiMax", a learnable piecewise softmax temperature meant to suppress the long tail of implausible tokens.
• ECHO instead of a multi-token prediction module: the backbone's own last layer predicts the extra future token, and the same layer doubles as the draft model for speculative decoding.
• Tokenizer: about 200K entries, with numbers split right to left in groups of three digits.

Training used the Muon optimizer and FP8 mixed precision. Baidu also describes FlashMask-4, an attention kernel for head dimensions up to 576/512 that it says is up to 175% faster forward and 74% faster backward than FlashAttention-4 MaskMod on SM100 (Blackwell) GPUs.

Benchmarks, all Baidu's own

Baidu compares against the closest open base models it could find: DeepSeek-V4-Flash-Base (284B, 13B active), Hy3-preview-Base (295B, 21B active) and MiMo-V2.5-Base (310B, 15B active), all run through the same harness. Selected results:

ERNIE leads on most of Baidu's 30-benchmark table, but not all of it: Hy3 preview is ahead on FullStackBench, Hy3 and MiMo on BBH with chain of thought, and DeepSeek on ChineseSimpleQA, INCLUDE-lite, HELMET-Rerank and InfiniteBench. A nine-point lead over DeepSeek on MMLU-Pro between base models of the same size is large enough to want an independent rerun before treating it as settled.

The license is not open source

The weights come under Baidu's own ERNIE 5.5 Flash Base Model License Agreement v1.0, not Apache or MIT. Non-commercial use is free. Commercial use is free up to US$50 million in annual revenue from the related products, but requires a one-time use notice to Baidu before you start and "Powered by ERNIE" attribution. Above $50 million you need Baidu's written authorization first. The revenue counted is the whole revenue of any product that uses the model, even if it also uses other models.

Section 5.5 also bars using the model, derivatives or their outputs "to improve any other artificial intelligence model", while distilling the model into a new one counts as creating a derivative, which stays under the same license. In practice, ERNIE outputs can train ERNIE derivatives but not your other models.