LLM Quantization and GGUF
Package low-bit weights and metadata into a deployable model artifact
Suggested reading: about 18 min
Learning goal
Chapter keywords
| Keyword | Explanation | ESP32 engineering analogy |
|---|---|---|
| GGUF | A model-container format used by GGML-family executors. | Like a firmware image with an extensible descriptor area. |
| Block quantization | Low-bit encoding in which values in a block share parameters such as a scale. | Like compressing a batch of samples and attaching that batch's range. |
| Weight-only | A strategy that quantizes weights while keeping activations at higher precision. | Like compressing firmware constants while preserving compute-workspace precision. |
| mmap | Mapping a file into virtual memory for on-demand access. | Like reading firmware resources by page to avoid copying everything at once. |
Bridge from the previous chapter
The previous chapter established a capacity ledger for KV state that grows token by token. This chapter compresses the weights reused across requests and treats GGUF metadata as a verifiable model-package contract.
Historical development
The history of low-bit models is not a story of simply truncating floating-point values into integers. Early compression research already treated pruning, quantization, and coding as separate layers. The LLM era then exposed outlier channels, layer sensitivity, and specialized-kernel requirements. GGUF addresses another dimension: reliably packaging tensors, quantization types, and the metadata needed to interpret a model into one quickly readable container.
Deep Compression places quantization in a systematic compression pipeline
Deep Compression combines pruning, post-training quantization, and Huffman coding, demonstrating that reduced bit width is only one stage of a compression pipeline. It also establishes an important engineering discipline: validate both task accuracy and actual storage savings after compression rather than reporting theoretical bit counts alone.
Original Deep Compression paper ↗LLM.int8 reveals the cost of large-model outlier features
LLM.int8 observes systematic large magnitudes in certain hidden dimensions and keeps those outlier computations on a mixed-precision path. This moves quantization beyond reducing every weight uniformly and explains why algorithms labeled int8 can have very different quality and kernel requirements.
Original LLM.int8 paper ↗GPTQ makes one-shot weight quantization practical for very large models
GPTQ uses approximate second-order information to quantize weights layer by layer, bringing large models into the 3- to 4-bit range without complete retraining. It strengthens the view that error must be compensated according to weight interactions and makes calibration samples and quantization order part of the deployment artifact.
Original GPTQ paper ↗GGUF unifies the tensor container and extensible metadata
As the successor to GGML, GGMF, and GGJT, GGUF reduces ambiguity through typed key-value metadata, an aligned tensor area, and explicit version fields, while supporting mmap access. It does not require one particular quantization; tensors of different types can coexist in the same container.
Official GGUF specification ↗Mixed-quantization recipes become target-hardware build steps
llama.cpp quantization tools support K-quants, importance matrices, and per-tensor type overrides. Practice moves from choosing one “Q4” label to retaining sensitive tensors, matching kernels, checking a perplexity proxy, and measuring on the target hardware. Requantizing a low-bit file is also explicitly treated as high risk.
Official llama.cpp Quantize documentation ↗Illustrated analogy
A compressed paint palette packed into a shipping container
The original FP16 weights resemble tens of thousands of subtly different studio paints. Quantization does not discard colors at random: it groups similar colors into small blocks and creates a finite swatch card for each block. Integer codes are swatch numbers, while scale and minimum are the restoration instructions. Sensitive layers, like a portrait's eyes, can retain a finer palette. GGUF is the shipping container with a manifest: its header records architecture, tokenizer, and alignment rules, while the cargo map says where every tensor lives and which encoding it uses so the runtime can move it to the correct kernel.
Where the analogy stops: The paint analogy omits matrix-multiplication kernels: a higher compression ratio does not automatically improve speed. Without a hardware implementation for the encoding, a runtime may dequantize before computing. Perceptually “similar colors” are not a measure of language-model quality either; perplexity or task sets, long-text stability, and target-device benchmarks still decide.
Chapter walkthrough
Calculate real BPW instead of trusting the Q4 name
In addition to low-bit codes, block quantization stores each block's scale, minimum, and alignment padding; some formats also mix tensor precisions. The file's average bits per weight therefore usually differs from the number in the name. Verify it by reading GGUF tensor types and element counts and summing data bytes and metadata overhead separately, then compare with an FP16 baseline. The capacity budget must also add the tokenizer, KV cache, and runtime workspace: model-file size is not peak RAM.
Block size controls compression and local adaptability
When a group of weights shares quantization parameters, a larger block lowers scale overhead but struggles to cover small values and outliers at once. A smaller block adapts more closely to local ranges but adds parameters, indexes, and kernel-processing cost. Per-tensor, per-channel, and block-wise are not abstract labels; they choose granularity between statistical fit and storage access. Inspect the exact target-format layout rather than forcing one generic formula onto every Q4 scheme.
Let sensitivity and kernels determine mixed precision together
Embeddings, output projections, attention, and some MoE tensors differ in their sensitivity to error; K-quant suffixes often denote a mixed recipe. Generate several candidates from the same high-precision GGUF, compare them on identical calibration samples and task sets, and confirm that the backend has efficient kernels for every tensor type in the recipe. If a few layers repeatedly fall back or dequantize, boundary conversion can erase the bandwidth savings.
GGUF is a self-describing container, not a quality certificate
A loader interprets a file from its magic, version, alignment, tensor shapes and types, and architecture metadata; tokenizer- and chat-template-related fields also determine input semantics. A valid format proves only that the bytes can be parsed, not that the weights have a trustworthy source, the quantization was correct, or the model output is sound. Preserve the source revision, converter commit, quantization command, and hash, and inspect critical metadata with a dump tool before deployment to prevent silent “loads correctly but answers consistently wrong” failures.
Build a one-way release pipeline for irreversible conversion
Quantization discards information. Requantizing Q4 into Q5 cannot restore precision and instead adds another round of rounding error. Keep trusted FP16/BF16 or official source weights and make conversion and quantization reproducible builds with a fixed input hash, fixed tool versions, and separately named outputs. Acceptance must measure file size, peak resident memory, PP/TG speed, task correctness, and long-output anomalies together so every regression can be traced back to a recipe.
Interactive process
How high-precision weights become a verifiable GGUF
The animation separates container conversion from tensor quantization so format, algorithm, and final deployment performance are not conflated into one step.
Freeze the baseline · BF16/FP16 weights + tokenizer + revision
Record source, license, hash, and baseline outputs
Every later low-bit artifact must trace back to the same high-precision starting point
Loop / return condition: If quality exceeds its error budget, return to “Collect sensitivity” and broaden representative samples or raise precision for sensitive tensors. If speed does not improve, return to “Quantize blocks” and inspect kernel support and dequantization boundaries. Always regenerate from the high-precision baseline; never chain requantization.
View the complete static diagram
FP16 tensor → blocks → quantized values + scales/mins → GGUF tensor + metadata → runtime kernel
Code or command example
python convert_hf_to_gguf.py ./model --outfile model-f16.gguf
llama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M
llama-cli -m model-q4_k_m.gguf -p "hello"Hands-on lab
Lab notes and export
Engineering pitfall
Knowledge check
Answer all three multiple-choice questions, then submit. Answers stay hidden until submission.
Discuss this chapter on GitHub
Sign in with GitHub to ask a question, share measurements, or compare implementations. Comments are stored in this course's GitHub Discussions.
The embedded comments need JavaScript. You can also open the GitHub discussion area directly: Open GitHub Discussions ↗
Further reading
Bridge to the next chapter
Next, the model package enters a real runtime and on-device framework, exposing loading, sampling, scheduling, backend partitions, fallback, and cancellation paths.
Next: Day 13 · LLM Runtime