sysnote
Analysis

llama.cpp FP8 has landed, but only per-tensor FP8 checkpoints stay FP8: we checked 13 of them

llama.cpp can now keep FP8 weights as FP8. Three pull requests by NVIDIA's Oliver Simons (ORippler) landed in the last three weeks: #27512 added an OCP FP8 E4M3 tensor type and taught the converter to keep FP8 tensors instead of dequantizing them (merged October 29), #28392 added FP8 KV caches (-ctk f8_e4m3 -ctv f8_e4m3, merged November 10), and #28413 added CUDA kernels that run FP8 matrix multiplies on FP8 tensor cores (merged November 11). All three are in llama.cpp v1.0.0, tagged on November 13. The release notes list the KV cache; the CUDA kernels are in the ggml v1.0.0 notes.

That sounds like "FP8 checkpoints from Hugging Face now just work". They mostly don't, yet, and the reason is in the converter: it keeps FP8 only when a tensor has a single scale. We checked how 13 public FP8 checkpoints store their scales. Seven would stay FP8. The other six, including Qwen's and DeepSeek's official FP8 releases, are still dequantized to 16-bit, as before.

Which checkpoints stay FP8

An FP8 E4M3 number has a 3-bit mantissa and tops out at 448, so FP8 checkpoints ship scale factors next to the weights. Their layout varies: one scale for the whole tensor, one per output row ("channel"), or one per 128×128 block. In conversion/base.py, the new _prepare_fp8_e4m3_tensors keeps a weight as F8_E4M3 only if its scale has exactly one element (prod(scale.shape) == 1). Everything else goes through the old dequant_fp8 path. FP8 is also kept only with the default --outtype auto; --fp8-as-q8 still turns FP8 weights into Q8_0.

We read the config.json and the safetensors header of the first shard of each checkpoint and matched the scale shapes against that rule. We didn't run the conversion, so this is a reading of the code rather than a test:

Checkpoint FP8 weight scales After convert_hf_to_gguf.py
nvidia/Qwen3-8B-FP8 (ModelOpt) per tensor, plus k_scale/v_scale stays FP8
nvidia/Llama-3.1-8B-Instruct-FP8 per tensor, plus k_scale/v_scale stays FP8
nvidia/Mistral-Medium-3.5-128B-NVFP4 (FP8 layers) per tensor, plus k_scale/v_scale stays FP8
nvidia/Qwen3.8-27B-NVFP4 (FP8 layers) per tensor stays FP8
RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8 per tensor stays FP8
RedHatAI/Llama-3.2-1B-Instruct-FP8 per tensor stays FP8
mistralai/Ministral-3-8B-Instruct-2512 per tensor (weight_scale_inv) stays FP8
RedHatAI/Qwen3.5-4B-FP8-dynamic per channel dequantized
unsloth/Qwen3.8-27B-NVFP4 (FP8 layers) per channel dequantized
Qwen/Qwen3-8B-FP8 128×128 blocks dequantized
Qwen/Qwen3.6-35B-A3B-FP8 128×128 blocks dequantized
deepseek-ai/DeepSeek-V3.2 128×128 blocks dequantized
MiniMaxAI/MiniMax-M2 128×128 blocks dequantized

The pattern is the producer. NVIDIA's ModelOpt and the older static Red Hat recipes use per-tensor scales, while the official FP8 releases of Chinese labs use DeepSeek's 128×128 blocks. Aman Gupta (am17an) raised exactly this in his review of #27512, calling 128×128 tile support what "would enable us to run 'true' inference for safetensors checkpoints for a lot of a Chinese OSS models". Simons deferred it to a later PR, probably as a separate block-scaled type. None is open yet.

What runs where

FP8 weights now work on every CUDA GPU, but they only run as FP8 math on some. From ggml-cuda/fp8.cu:

  • Prefill and other large batches go through cuBLASLt FP8 GEMM, but only when fp8_mma_hardware_available() is true: NVIDIA compute capability from Ada Lovelace (8.9) up to, but not including, Rubin. That covers RTX 40 and 50, L40S, H100 and B200. Activations are quantized to FP8 on the fly with one scale for the whole activation tensor, its absolute maximum divided by 448.
  • Small batches (eight tokens or fewer, which covers single-user decoding) use the MMVF kernel. It converts FP8 weights to floating point and multiplies in F32. Token generation reads half the bytes of BF16, but doesn't use FP8 math.
  • Older GPUs fall back to cuBLAS with the weights converted to FP16, so an RTX 3090 loads these files and runs them like a 16-bit model.
  • Elsewhere, the CPU has a scalar reference implementation. Metal explicitly reports FP8 matrix multiplies and row lookups as unsupported, so on a Mac those operations fall back to the CPU. Vulkan support is a separate draft, #29791 by Ruben Ortlam (0cc4m). The PR also defers dedicated FP8 MoE kernels, on the reasoning that "folks tend to quantize those layers to 4/2-bit".

Simons's own numbers are for the FP8 layers of nvidia/Qwen3.8-27B-NVFP4 on an RTX PRO 4500 Blackwell. With those layers kept in FP8 (19.56 GiB file), llama-bench measured 3,582 tokens/s on an 8,196-token prompt. With the same layers as Q8_0 (19.98 GiB), it measured 3,063 tokens/s. That makes prefill 17% faster, while generation was flat at about 40 tokens/s either way. Wikitext-2 perplexity was 5.4527 with FP8 layers and 5.4727 with Q8_0 layers, both with a BF16 KV cache. These come from one GPU, one model and the author's own early build, measured on just 10 chunks. They suggest more than they prove.

The FP8 KV cache is barely smaller than Q8_0

-ctk f8_e4m3 -ctv f8_e4m3 stores each cached key and value in one byte. llama.cpp's existing Q8_0 cache uses 34 bytes per 32 values, one FP16 scale plus 32 int8s, or 1.06 bytes per value. The FP8 cache saves about 6% of KV memory compared with q8_0, and half compared with F16. In the same perplexity table it was a little worse than Q8_0: 5.4550 against 5.4491, with BF16 at 5.4527.

So for an ordinary GGUF model, -ctk q8_0 -ctv q8_0 is still as good a choice. The point of the FP8 cache is checkpoints that ship calibrated KV scales. ModelOpt's nvidia/Qwen3-8B-FP8 and nvidia/Mistral-Medium-3.5-128B-NVFP4 both have per-layer k_scale and v_scale. #28392 converts them into the GGUF and applies them in the attention graph, so llama.cpp now uses the same calibration as vLLM or TensorRT-LLM instead of assuming a scale of 1. A few practical details:

  • The default CUDA build compiles flash-attention kernels only for the f8_e4m3/f8_e4m3 pair. Mixing an FP8 K cache with a Q8_0 V cache needs GGML_CUDA_FA_ALL_QUANTS or a custom GGML_CUDA_FA_QUANTS list. FP8-native attention in the tile kernels used for prefill is deferred to a follow-up.
  • The cache refuses to start on a list of architectures whose attention reads the KV cache directly: every MLA model plus DeepSeek 3.2/4/4.1, GLM-DSA, GLM-5-next, dots3note, Mistral Large 4, Hy V4, MiniMax M3, qwen4exp and DFlash. You get FP8 cache is not supported for architecture .... Simons wrote that MLA models mostly use dynamic block scales anyway. His follow-up for DFlash and MLA, #31737, was closed unmerged, and the list is unchanged on master as of November 16.

What #28898 is for

The open piece is #28898, "ggml: support quant scales for fp8 and nvfp4" by Ortlam, still a draft as of November 16 (2,805 lines added, 2,511 removed, 209 files). Right now, per-tensor weight scales are applied by llama.cpp's model code as a separate multiply in the graph, which CUDA then fuses. Each model file has to remember to do it. #27512 itself says the stack "does not add plumbing of scales" and that #28898 "will make misuse of scaled types error at least".

#28898 moves the scales into ggml: ggml_mul_mat and ggml_mul_mat_id get an optional weight scale and activation scale for the types that need them. Ortlam argues this also "permits optimizations for src1-quantization through preset activation scales that models ship with". Those would replace the absolute-maximum pass the CUDA path now runs before every FP8 GEMM. Georgi Gerganov approved the direction in September, and on November 11 Ortlam wrote he would "update this soon and get it ready". It touches every model file and breaks the current NVFP4 scale path, so it will take a while to land.

For now the useful summary is short. Per-tensor FP8 checkpoints, mostly NVIDIA's, can become FP8 GGUFs that run on FP8 tensor cores on Ada and newer GPUs. Block-scaled FP8 from Qwen, DeepSeek and MiniMax still becomes BF16 on the way in. For KV memory, Q8_0 already got you almost all of the way.

Comments

No comments yet. Be the first.

Write a comment

Comments are published after review.