Model Quantization
Trade fewer bits for lower storage and bandwidth
Suggested reading: about 19 min
Learning goal
Chapter keywords
| Keyword | Explanation | ESP32 engineering analogy |
|---|---|---|
| Scale | The proportional factor between an integer code and its real value. | Like an ADC range and the physical meaning of one code step. |
| Zero-point | The integer offset that represents real zero in the mapping. | Like the reference code after correcting a sensor’s zero offset. |
| Calibration set | Representative input samples used to estimate activation ranges. | Like calibrating an ADC with real operating conditions instead of one midpoint input. |
| Per-channel | Using separate quantization parameters for different weight channels. | Like calibrating gain independently for each sensor channel. |
Bridge from the previous chapter
The previous chapter established delivery contracts from training state to model package. This chapter changes numeric representation, exchanging measured error for storage, bandwidth, and kernel opportunities.
Historical development
Neural-network quantization did not simply appear in response to mobile chips. Early digital and analog neural hardware already faced finite word lengths. As deep learning enlarged models, research moved from asking whether finite precision could work to co-designing compression, low-bit training, and integer operators. Today’s PTQ, QAT, per-channel methods, and representative calibration are engineering outcomes of decades spent balancing precision, storage, and hardware.
Finite word length became a neural-computing system design problem
Neural Network Number Systems compared fixed-point, floating-point, and logarithmic number representations for digital neural networks. Early work already recognized that bit width affects circuit cost, dynamic range, and computational error together; it is not merely a change of file dtype.
Record of the original Neural Network Number Systems paper ↗Deep Compression placed quantization in a complete model-compression pipeline
Deep Compression combined pruning, post-training weight-sharing quantization, and Huffman coding, demonstrating substantial reductions in model storage and memory traffic. It reinforced an engineering fact: a compression method must be evaluated together with encoding overhead, its decode path, and target hardware.
Original Deep Compression paper ↗Low-bit weights and activations entered the training process
Quantized Neural Networks introduced low-precision weights and activations into forward and backward computation and studied extremely low-bit representations. Quantization was no longer packaging applied after training; training could expose its error explicitly so parameters adapted to a discrete codebook.
Original Quantized Neural Networks paper ↗Integer arithmetic ran from input through output
Jacob and colleagues presented a quantization scheme and training flow for integer-arithmetic inference, quantizing weights and activations while handling scale factors, zero-points, and accumulation. Algorithm design explicitly connected to ARM CPUs and integer accelerators instead of reporting only a smaller file.
Original CVPR integer-inference paper ↗PTQ calibration became a standard stage in edge toolchains
LiteRT full-integer quantization uses representative data to estimate the ranges of inputs and intermediate activations, and can make model I/O integer as well. Current practice therefore emphasizes calibration data, operator coverage, and measurement on target hardware rather than treating an int8 label as a performance guarantee.
Official LiteRT full-integer quantization tutorial ↗Illustrated analogy
Printing a continuous mountain scene with a limited set of tones
A photographer has an original rich in subtle gradations, but the press offers only a finite number of levels for each color. The plate maker studies the darkest and brightest areas across the whole collection before choosing the range covered by one tone. Colors between levels must be rounded, while highlights and shadows outside the paper’s range collapse to the same extrema. Separate color plates resemble choosing an independent range for each channel.
Where the analogy stops: Printed tonal differences are judged visually, whereas model error accumulates through many operators and must be measured by task metrics, not by whether it “looks similar.” Real integer kernels also involve bias scales, int32 accumulation, requantization, and hardware instructions that the printmaking analogy cannot explain.
Chapter walkthrough
Start with the approximately reversible formula to expose three error entry points
Affine quantization uses q=round(x/scale)+zero_point and then clamps q to the integer range; dequantization recovers only an approximation on that grid. Error comes from rounding at finite spacing, saturation when the range is too narrow, and wasted resolution when outliers stretch the statistical range. Measure quantization MSE, lower/upper-bound hit rates, and the zero mapping separately. Folding them into one accuracy number makes diagnosis difficult.
Symmetric, asymmetric, and per-channel schemes are backend contract choices
Symmetric quantization usually sets zero-point to zero for a simpler multiply path; asymmetric quantization uses more of the integer range for skewed distributions. Per-channel quantization assigns a scale to each output channel and often handles cross-channel differences in weight magnitude better, but its kernel must know the quantization axis and load multiple parameter sets. Before choosing, inspect the target runtime’s dtype, axis, and operator constraints rather than relying on PC accuracy.
Calibration is not sampling a few “typical images”; it must cover activation state space
Weight ranges can be scanned directly, but intermediate activations depend jointly on inputs and preceding layers. Representative data should span lighting, silence, saturated sensors, class boundaries, and device noise, entering through the real preprocessing path. Inspect per-layer histograms and clipping rates to see whether a few outliers dominate the scale. If the production distribution changes, treat the old calibration table as a versioned asset that needs revalidation.
Integer convolution still needs wide accumulation and correct scale handoffs
Products of int8 inputs and int8 weights usually enter a wider accumulator. The bias scale should align with input scale × weight scale, and the output is requantized using a multiplier and shift or an equivalent operation. Any difference in zero-point compensation, rounding rule, or saturation order can introduce systematic bias. Hand-calculating accumulation and boundary values for a tiny matrix is one of the most effective unit tests for a custom kernel.
Put accuracy regression and performance acceptance in one layer-level report
First compare final task metrics between float and quantized models. For samples with the largest output drift, locate per-layer SQNR, cosine similarity, or maximum absolute error; at the same time, inspect the profile for Quantize/Dequantize islands and float fallback. If the file becomes smaller but execution slower, the cause may be missing integer kernels, layout conversion, or frequent rescaling—not a failure of quantization itself.
Interactive process
How a floating-point waveform lands on 256 integer rungs
The animation first stretches a ruler over measured data, then snaps each floating-point value to its nearest rung. Out-of-range points hit red barriers before the integer stream enters an accumulator.
Scan the distribution · activation samples → min/max/histogram
Spread samples into a histogram and color the long tail separately
Activation ranges must be estimated from representative real inputs
Loop / return condition: If error or saturation exceeds the limit, return to distribution scanning and change calibration samples, quantization axis, or range strategy. If performance misses its target, return to the integer kernel and inspect fallback and conversion boundaries.
View the complete static diagram
float x ──(x/scale)+zp──► int8 q ──runtime kernel──► int8/float output
Code or command example
q = round(x / scale + zero_point)
q = clip(q, -128, 127)
x_hat = (q - zero_point) * scaleHands-on lab
Lab notes and export
Engineering pitfall
Knowledge check
Answer all three multiple-choice questions, then submit. Answers stay hidden until submission.
Discuss this chapter on GitHub
Sign in with GitHub to ask a question, share measurements, or compare implementations. Comments are stored in this course's GitHub Discussions.
The embedded comments need JavaScript. You can also open the GitHub discussion area directly: Open GitHub Discussions ↗
Further reading
Bridge to the next chapter
Next, operator contracts lead into layouts, tiling, fusion, and hardware kernels.