OPTIC shrinks Gemma 4's per-layer embeddings about 100x by giving each token one code for all layers

Gemma 4's small models carry an unusual amount of baggage. In E4B, the per-layer embeddings (PLE), a lookup table that gives every vocabulary item its own 256-wide vector at every decoder layer, account for 35.2% of the parameters and 5.64 GB in BF16. In E2B the share is 45.8%. A paper posted on October 22 by researchers at MBZUAI, the Weizmann Institute and elsewhere squeezes the E4B table to 56.4 MB, roughly 100 times smaller, and reports that the model keeps 97.8% of its benchmark performance.

The interesting part is not the ratio but where the redundancy turned out to be.

One code per token, not per layer

At around 100x, the E4B table's 11 million token–layer vectors get about five bytes each. Conventional quantizers spend those bytes vector by vector, and at this budget they fall apart: product quantization, the strongest of six baselines the authors adapted to PLE, kept 66.3% of performance on BF16 E4B. On GSM8K it dropped from 92.9% to 73.5%, and on the IFBench instruction-following test from 44.0% to 16.0%.

The method, called OPTIC, stops treating the 42 layers' vectors for a token as separate objects. Each token gets a single code tuple, 161 one-byte codebook indices that together select a 644-dimensional latent vector, and each layer has its own affine decoder that turns that shared latent into the layer's PLE row. Because every token in a layer shares the same decoder, decoder storage grows with the number of layers but not with the vocabulary.

The second ingredient is what the fit optimizes. PLE vectors reach the model only through a gate and an output projection, so OPTIC minimizes error after that projection rather than in the stored coordinates. The paper's analysis shows why this matters: in stored coordinates, 212 to 227 of 256 principal directions are needed to capture 90% of a layer's token-specific variation, but measured at the projection output the figure ranges from 1 to 205 depending on the layer.

The ablations support the sharing idea specifically. With the same fitting recipe and nearly the same storage, giving every token–layer pair its own latent code retained 79.3% on BF16 E4B, against OPTIC's 97.8%. Throwing the token-specific information away entirely, by replacing every vector with its layer's average, cut GSM8K to 43.5% and AIME 2026 to zero, so the table is not dead weight that any approximation could stand in for.

What survives and what doesn't

The 97.8% figure is a geometric mean over seven tasks, with each task's retention capped at 100% so that a gain on one cannot hide a loss on another. Per task on BF16 E4B, GSM8K went from 92.9% to 92.8%, GPQA Diamond from 56.9% to 54.7%, MMLU-Redux from 78.9% to 76.6%, IFBench from 44.0% to 41.7% and InfographicVQA from 69.7% to 67.5%. AIME 2026 rose from 45.8% to 48.3%, which likely says more about the noise in a 30-problem test than about compression; the authors themselves warn that a single low-count task can swing the aggregate.

The smaller model is less forgiving. E2B retained 89.1% (90.0% with a variant that gives frequent tokens more code groups), and its knowledge scores took the visible hit: MMLU-Redux fell from 71.5% to 62.5% and GPQA Diamond from 43.2% to 32.2%. When PLE is nearly half the model, a lossy PLE costs more.

The method also stacks with Google's own 4-bit release. On the official W4A16 checkpoints, whose PLE lookup stays in BF16, OPTIC retained 96.9% on E4B and 91.5% on E2B. Pushed harder on BF16 E4B, to about 594x and 1,002x, it still kept 90.5% and 84.1%.

Where the memory goes

Served with vLLM on a single RTX 4090 with a fixed 1 GiB key–value cache, BF16 E4B used 13.05 GB with compact PLE instead of 18.57 GB, 29.7% less. For the W4A16 model the drop was 43.1%, from 12.95 GB to 7.37 GB, because once the backbone is quantized, the BF16 embedding table is the largest thing left. Decoding rows on the fly added at most 0.40% to wall time for BF16 and 0.69% for W4A16 across the authors' eight workloads.

The freed memory matters more as cache than as a number. With both versions given the same 21.6 GiB budget, vLLM assigned 4.87 GiB to the key–value cache with dense PLE and 10.06 GiB with compact PLE. Sixteen concurrent 32K-token requests finished in 78.7 seconds instead of 105.1. The dense model completed an 88,000-token prompt, but a 96,000-token one was accepted and never scheduled within 15 minutes; the compact version handled 130,000 tokens in 33 seconds. At 16 GiB, the dense BF16 model would not start at all.

There is a fair objection here. Google designed PLE to be kept out of accelerator memory: its Gemma 3n documentation, where PLE first appeared, describes caching the parameters to fast local storage and adding them as each layer runs. A runtime that does this already avoids much of the GPU cost. The paper tests that case too, and the compact table still helps, because it shrinks what has to cross the bus. Fetching the PLE rows for 8,192 tokens from pinned host memory took a median 18.6 ms dense and 1.5 ms compact. For a single token, dense was faster, 27 against 46 microseconds, so the gain shows up in batched lookups such as prefill (compact was faster from 32 tokens up), not in token-by-token decoding.

Caveats

This is one paper on one model family, and we found no independent reproduction yet. The main comparisons use a single fitting seed, so the reported confidence intervals (95.5–99.0% for OPTIC on BF16 E4B) describe benchmark sampling, not variation between fits. The baselines are the authors' adaptations of general-purpose quantizers to a table they were not designed for. The serving numbers come from one consumer GPU and vLLM 0.19.1, not from the on-device runtimes Gemma's small models are aimed at.

The code is on GitHub and a compressed E4B checkpoint is on Hugging Face, so checking it is cheap.

The authors frame PLE as one example of a broader move toward bigger lookup tables, alongside DeepSeek's hash-addressed Engram memory and the 51B parameters of n-gram embeddings attached to Qwen3.8-Flash-Next. Whether those tables compress the same way is untested. For Gemma, at least, much of what looks like per-layer information can be rebuilt from one token representation passed through 42 different decoders, which is why a per-vector quantizer was the wrong tool for it.