Training and Inference
Freeze learnable state into a portable, verifiable model artifact
Suggested reading: about 20 min
Learning goal
Chapter keywords
| Keyword | Explanation | ESP32 engineering analogy |
|---|---|---|
| eval mode | A mode that makes Dropout and BatchNorm use inference semantics. | Like switching from a debug configuration to a production configuration. |
| Optimizer | An algorithm—and its state—that updates parameters from gradients. | Like an automatic tuning controller that should not remain in deployed firmware. |
| Computation graph | A set of nodes and edges describing how tensors pass through operators to produce outputs. | Like a firmware data-flow graph or task-dependency graph. |
| ONNX opset | The version of operator semantics declared by a model. | Like a communication-protocol version: sender and receiver must be compatible. |
Bridge from the previous chapter
Day 1 explained tensors, gradients, and one weight update. This chapter follows that mutable training state as it is frozen, converted, and delivered to a device runtime that never executes backpropagation.
Historical development
The model-delivery chain formed from training algorithms, execution modes, graph representations, and interoperability standards. It advances “one reproducible experiment” into “one artifact preserving semantics in another runtime.”
The perceptron learning rule formed a minimal training loop
The perceptron changed connection weights in response to prediction errors, closing the loop among samples, predictions, errors, and updates. Although the model was simple, it already separated the moment when examples modify parameters from the moment when current parameters produce a decision.
Original perceptron paper by Rosenblatt ↗Dropout clearly exposed two execution semantics for one module
During training, Dropout randomly removes units to reduce co-adaptation; at test time it uses a deterministic approximation of the full network. Engineers consequently had to switch modes explicitly, or identical inputs would receive unstable outputs from training-time randomness.
Original Dropout paper ↗Lightweight runtimes narrowed the device role to low-latency forward execution
The TensorFlow Lite developer preview targeted mobile and embedded devices, emphasizing a small footprint, fast initialization, and hardware acceleration. Converting models from a training framework into a constrained deployment format gave the toolchain explicit responsibility for separating training state and debugging convenience from device runtime constraints.
Official TensorFlow Lite announcement ↗Caffe promoted model reuse through declarative networks and weight files
Caffe organized network structure, training configuration, and learned parameters as exchangeable artifacts, using layers as extension units. It demonstrated that a model could be shared and deployed outside one research script, while also exposing the problem of framework-specific layers tied to a format.
Original Caffe paper ↗ONNX v1 published interoperability as a graph-and-operator contract
ONNX v1 released a production-ready format for multiple frameworks, promoting model transfer through an open graph format and operator sets. Exporters, converters, and runtimes could now collaborate around a common IR, although each still had to support the domains and opsets declared by a model.
Official ONNX v1 release announcement ↗GGUF emphasized single-file packaging, extensibility, and fast loading
GGUF, a model format for GGML-family runtimes, placed tensors and key-value metadata in an extensible binary container and superseded earlier GGML, GGMF, and GGJT formats. It showed that LLM deployment must preserve not just weights, but also reliable interpretation data such as architecture and tokenizer metadata.
Official GGUF specification ↗Illustrated analogy
From wind-tunnel trials to flight-control software sealed into a drone
Training resembles repeated wind-tunnel flights: engineers change parameters, inject disturbances, record each deviation, and tune the control law. Inference resembles a production drone taking off. It carries only the finalized control tables, sensor interfaces, and required state—not the entire wind tunnel, trial logs, and tuning team. Export is the certification step that turns an experimental configuration into a flashable release.
Where the analogy stops: Flight control is usually built from explicit physical models and safety arguments, whereas a neural network is statistical and cannot be assumed to cover every field condition merely because it is “finalized.” Nor does the analogy mean the graph stops changing after export: constant folding, quantization, and backend partitioning still alter numerical and resource behavior.
Chapter walkthrough
A training loop advances both parameter state and optimizer state
A step is more than a forward pass followed by subtracting gradients. Optimizers such as Adam maintain historical statistics for every parameter, and learning-rate schedulers have their own step counts. Resuming from weights alone while omitting optimizer state and epoch changes the training trajectory; none of that state is useful for pure inference. Treat “resumable checkpoint” and “deployable weights” as two separately named, validated, and archived artifacts.
Verify mode switches module by module; do not trust one Boolean
Calling eval recursively switches the behavior of modules such as Dropout and BatchNorm, but a custom layer may still read a training flag or use random numbers. BatchNorm running means and variances may also be distorted by small batches or data drift. Before export, run the same input repeatedly to confirm deterministic output, then inspect random operators, statistics buffers, and requires-grad state individually.
A deployment graph is a transformed artifact, not a photocopy of the checkpoint
An exporter may inline functions, remove backward nodes, fold constants, merge Conv and BatchNorm parameters, or specialize control flow using an example input. These changes reduce scheduling and memory overhead, but may capture only the branch exercised by the example. If a model contains data-dependent loops, dynamic shapes, or custom operators, define the export strategy explicitly rather than assuming every source-program path was preserved.
Separate a format into syntax, semantics, and companion assets
A schema only says how fields are encoded; operator specifications say what nodes must compute; tokenizers, label maps, and normalization parameters may live in metadata or external files. Successful parsing proves only that the syntax is valid. Missing operator semantics makes a runtime reject the model, while missing assets may let it run but misinterpret its outputs. List validation methods for every layer in the delivery checklist instead of recording only one model-file hash.
IR, opset, and runtime versions are three different locks
The IR version constrains the model container and graph structure, an opset constrains signatures and semantics in a particular domain, and the runtime version determines implementation coverage. Lowering an opset is not changing an integer: if the older set has no equivalent expression, the converter must decompose a node or fail. A custom domain also requires shipping its implementation. A compatibility matrix should record all three versions with the target chip, not merely say “supports ONNX.”
Converters rewrite representations, so audit the change list
Conversion may expand a high-level operator into a subgraph, precompute constants, change NCHW to NHWC, or store weights in another dtype. Even mathematically equivalent rewrites can alter rounding, workspace, and backend partitioning. Preserve counts of added, removed, and replaced nodes in conversion logs, and map names for important intermediate tensors so an abnormal final output does not leave you guessing.
Build a model ABI test bundle instead of saving the model alone
Attach input names, shape ranges, dtypes, units, preprocessing and postprocessing versions, and several golden samples to every model version. Samples should include normal values, boundary shapes, and invalid inputs. CI first runs schema/checker validation, then loads the model in the target runtime and compares outputs within tolerance. If the model uses external weights or tokenizer files, validate relative paths and hashes as well, preventing a stable main file from hiding drift in companion assets.
Interactive process
A model passes through five format gates
The same test data travels with the model from its training framework to the target backend. Each gate checks one type of contract, and any red light stops the journey near its root cause.
Framework object · modules + parameters + control flow
Expand the modules actually executed and their parameter references
The source program may contain dynamic behavior that cannot be serialized directly
Loop / return condition: When a runtime, opset, converter, or companion asset changes, replay from the affected gate. Golden inputs always travel with the model, forming a regression-ready format passport.
View the complete static diagram
data + labels → train state → checkpoint
│ eval/freeze/export
▼
input contract → graph + weights + metadata → runtime/backendCode or command example
model.eval()
export(model, example_input, opset_version=...)
assert_close(reference_output, runtime_output)
# Preserve tokenizer/labels/preprocess/version tooHands-on lab
Lab notes and export
Engineering pitfall
Knowledge check
Answer all three multiple-choice questions, then submit. Answers stay hidden until submission.
Discuss this chapter on GitHub
Sign in with GitHub to ask a question, share measurements, or compare implementations. Comments are stored in this course's GitHub Discussions.
The embedded comments need JavaScript. You can also open the GitHub discussion area directly: Open GitHub Discussions ↗
Further reading
Bridge to the next chapter
Next comes quantization: after fixing the model ABI, calibration and error evidence map floating-point values into fewer bits.