LFM3 keeps 128K tokens of context in 1.6 GB, and its benchmark lead depends on a score for abstaining

Liquid AI's new LFM3 family is built around a 1.7-billion-parameter text model that does something unusual for its size: it keeps a 128K-token context in about 1.6 GB of KV cache. The architecture is the interesting part of the October 27 release. The benchmark table that comes with it is interesting too, though partly for a reason Liquid doesn't point out: its headline average is decided by a benchmark that rewards a model for declining to answer.

The release covers six models of roughly 2B parameters, all branched from one backbone, LFM3-1.7B-Base: a text model, a vision model (LFM3-VL-2B), an audio model, an "omni" model that combines vision, audio and speech output, and two "decision" models (d1-2B and d1-Omni-2B) that answer a list of questions about an input in a single forward pass. The weights are on Hugging Face under Liquid's LFM Open License v1.0, which is not an OSI license: commercial use is licensed only for organizations with less than $10 million in annual revenue.

Six attention layers out of 24

LFM2 mixed short convolutions with attention. LFM3 adds a third ingredient. According to the model's config.json, the 24 layers are 13 gated short convolutions (kernel size 7, up from 3), 5 sliding-window attention layers with a 256-token window, and 6 global attention layers. Liquid says the mix came out of its hardware-in-the-loop architecture search, which this time measured candidates on 14 device and backend combinations, from a Raspberry Pi 5 and two Android phones to Arm and x86 cloud CPUs, GPUs and a Qualcomm Hexagon NPU.

That layout is what makes the long-context memory claim work. Only the 6 global layers have to keep keys and values for every token; the sliding-window layers keep 256 tokens and the convolutions keep a few positions of state. With 4 KV heads of dimension 128 in BF16, the global layers need 6 × 4 × 128 × 2 (keys and values) × 2 bytes = 12,288 bytes per token. Multiply by 131,072 tokens and you get 1.61 GB, which matches the 1.6 GB Liquid reports, against 4.3 GB for Qwen3.5-4B and 5.6 GB for MiniCPM5-2B in its own comparison.

A few other choices are worth noting. Global attention layers apply RoPE to only half of each head (partial_rotary_factor: 0.5), leaving the low-frequency half unrotated so it matches on content regardless of distance; the sliding-window layers keep full RoPE. A sigmoid gate on each head's output lets a head switch itself off, which Liquid says helps with the activation outliers that hurt quantization. Pre-training ran to 22T tokens, with a late mid-training stage heavy on agentic traces and tool calls, then context extension to 256K.

The trade-off shows up where you would expect it. On RULER at 128K tokens, LFM3-1.7B scores 65.56; Qwen3.5-4B scores 84.63 and Qwen3.5-9B 87.11. Cheap long context and accurate long context are not the same thing, and six global layers are not many. That is still a big jump from LFM2.5-2.6B, which scored 1.15 on the same test.

What the average is made of

Liquid's model card compares LFM3-1.7B with LFM2.5-2.6B, MiniCPM5-2B, Gemma 4 E4B, Qwen3.5-4B and Qwen3.5-9B on 29 benchmarks, all run by Liquid with vLLM in BF16 at the same sampling settings. The bottom line is an unweighted average: 49.37 for LFM3-1.7B against 47.24 for Qwen3.5-9B, a model about 5.7 times its size.

Three of those 29 rows come from AA-Omniscience, Artificial Analysis's knowledge benchmark. Its index rewards correct answers, penalizes wrong ones and, in Artificial Analysis's words, "has no penalty for refusing to answer." LFM3-1.7B has the lowest accuracy of the six models on it (7.07%), but because it rarely guesses, its index is 0.50 and its "non-hallucination" score is 92.93. Qwen3.5-4B, which has the best accuracy in the table (15.67%) but answers more often when wrong, gets an index of −57.17 and a non-hallucination score of 13.64.

Calibration is a fair thing to measure, and a small model that knows what it doesn't know is useful. But the same behavior is counted twice in the average, and it is worth a lot of points. Recomputing from the published table:

Take out the two abstention-sensitive rows and Qwen3.5-9B is slightly ahead; take out all three and the two are tied. LFM3-1.7B still beats Qwen3.5-4B comfortably either way, which is a good result for a model with less than half the parameters. "Competes with models several times its size," as Liquid puts it, is defensible. "Beats them on average" depends on the scoring.

The row-by-row picture is more informative than any average. LFM3-1.7B leads the 9B model on instruction following (IFEval, IFBench, IFStruct, Multi-IF), on function-calling tests like BFCL V4 and ToolSandbox, and on the long multistep browser and business-app tasks: 38.42 on WebArena-Verified against 28.94, and 41.29 on AutomationBench against 15.10. It trails on knowledge (63.37 on GPQA Diamond against 80.40), on agentic coding (43.40 on SWE-bench Verified with OpenCode against 51.20) and on long-context retrieval. That is roughly the profile Liquid describes, a small model post-trained hard for agent harnesses rather than a small model that knows more.

All of these numbers are Liquid's own, run with one shared vLLM configuration (temperature 0.6, top_p 0.95) that may not match each competitor's recommended settings. The on-device speed comparisons have a similar caveat: LFM3 ran on a llama.cpp build with LFM3 support, the other models on release b11744. Support for LFM3-1.7B and LFM3-VL-2B has since been merged into llama.cpp (PR #31252), so independent speed and quality checks should be easy to run. Until they appear, the architecture and the memory arithmetic are the verified part of this release; the leaderboard is a claim.