Edge Model Deployment
Put non-LLM vision, audio, and sensor models onto a real device
Suggested reading: about 19 min
Learning goal
Chapter keywords
| Keyword | Explanation | ESP32 engineering analogy |
|---|---|---|
| Representative data | A dataset covering real device operating conditions. | Like system-level tests spanning temperature, humidity, power, and load boundaries. |
| Tensor arena | A memory region preallocated for MCU inference inputs, outputs, and intermediate tensors. | Like a static workspace planned at startup. |
| Preprocessing | The steps that transform raw sensor data into model input. | Like framing, byte order, and validation in a protocol stack. |
| Confusion matrix | A table counting predicted classes against true classes. | Like counting false alarms and missed detections by fault category. |
Bridge from the previous chapter
The first four chapters established model math, artifact contracts, quantization, and kernels. This chapter puts them into a real TinyML training-to-firmware loop.
Historical development
Putting a small model on an MCU is not a scaled-down copy of desktop inference code. It followed a shared evolution toward architectures requiring less compute, kernels fitting hardware more closely, runtimes with fewer dependencies, and repeatable product validation—eventually forming today’s closed loop from data to firmware.
Structured convolutional networks reached real tasks
LeCun and colleagues applied local connectivity, weight sharing, and backpropagation to postal-code recognition. They showed that networks could process pixels directly while structural constraints reduced parameter and compute requirements.
Original paper by LeCun and colleagues ↗MobileNet put the hardware budget into network design
MobileNet used depthwise separable convolutions plus width and resolution multipliers to trade accuracy against latency explicitly. “Set the device budget before selecting a model” moved from an experienced habit to a tunable design method.
Original MobileNet paper ↗Production ML began to emphasize system testing
Google’s ML Test Score decomposed data, features, models, infrastructure, and monitoring into concrete test items, showing that offline metrics are only one part of production readiness.
Original Google Research paper ↗CMSIS-NN brought quantized models close to Cortex-M
CMSIS-NN supplied optimized Cortex-M kernels for convolution, fully connected layers, and more, using techniques such as partial im2col to reduce both runtime and peak memory.
Original CMSIS-NN paper ↗TFLite Micro established a microcontroller runtime pattern
TFLite Micro targeted resource-constrained, fragmented platforms without virtual memory. A small interpreter and preplanned workspace allowed one model to enter many kinds of MCU.
Original TFLite Micro paper ↗ESP-DL connects quantization, formats, memory planning, and analysis
Current ESP-DL combines ESP-PPQ, the .espdl format, static memory planning, and on-device profiling in one toolchain for model conversion, loading, execution, and measurement.
Official ESP-DL introduction ↗Illustrated analogy
A miniature factory turning fresh fruit into juice
Fruit arriving from the orchard varies in size, ripeness, and dirt. The lab first defines acceptance samples and a washing recipe, then trains an “inspector.” The production line must cut and weigh fruit according to exactly the same recipe, and inspection results still pass through thresholds and review before the bottling valve actually opens.
Where the analogy stops: The analogy explains a closed loop and shared contracts, but a model is not a fixed-rule inspection machine: it outputs probabilities or scores that depend on the data distribution. When a new field condition was absent from training data, a perfectly matched recipe still cannot guarantee correctness; drift monitoring, rejection, and a safety state machine remain necessary.
Chapter walkthrough
Work backward from the product event to the data, not forward from a convenient dataset
First write down the event the device must publish, tolerable false negatives and false positives, the response deadline, and what happens when no decision is possible. Use those requirements to define sampling windows and labels. Split training data by device or collection batch so adjacent portions of one continuous signal cannot leak into both training and test sets. Negative classes need more than “quiet background”; include confusing actions, noise, and sensor faults. Preserve firmware version, sample rate, and environment metadata on every sample so regressions can locate data drift.
Turn preprocessing into a verifiable binary contract
The PC training pipeline and firmware must agree item by item on sample rate, channel order, crop window, interpolation, color space, normalization constants, rounding, and saturation. Do not compare only the final class. Select golden samples and calculate summaries and spot-check elements at three boundaries: raw buffer, preprocessed float tensor, and quantized integer tensor. If the deployment tool expects int8 input, confirm the direction of scale and zero-point so already-quantized data is not quantized again.
Treat export and conversion as a compiler chain, not Save As
A training checkpoint first switches to inference semantics and exports a graph with a fixed I/O contract. The converter then folds constants, replaces operators, quantizes, and produces a target format. Every stage may change layout or values, so pin tool versions, save operator lists, I/O names, and quantization parameters, and compare the same inputs stage by stage. Successful conversion proves only that a file can be produced; the target can still fail from a missing kernel, float fallback, or an out-of-range shape.
Make inference a firmware task governed by scheduling constraints
The device must schedule sensor producers, DMA, preprocessing, inference, and event consumers together. Use bounded queues with an explicit full-queue policy: dropping old frames is usually more appropriate for real-time work than waiting forever. If inference occupies the CPU for too long, partition it, reduce frequency, or service the watchdog—but never use watchdog feeding to hide a deadlock. Allocate arenas, the model, and I/O buffers during initialization where possible, avoid fragmentation in the steady-state path, and make every error branch return its frame and publish an observable degraded state.
Complete deployment acceptance with four layers of evidence
Compare the source framework, exported model, target runtime, and physical board in sequence, using fixed golden vectors and full replays of raw captures. Results should include at least a confusion matrix, per-class threshold curves, p50/p95 latency, peak arena use, long-running tests, and abnormal inputs. At the business layer, verify hysteresis, consecutive-hit counts, and cooldowns so scores near a threshold do not become an event storm. Rerun the same checklist after every model or SDK update to separate accuracy regressions from system regressions.
Interactive process
How one sample becomes a device event
The player illuminates each data form and responsibility boundary, then returns field failures to the dataset at the end.
Collect and label · raw window + label + metadata
Record raw samples by device and scenario, then freeze labeling rules
The source data defines the world the model can recognize
Loop / return condition: After human review, field samples reenter “Collect and label,” while the old test set remains frozen to determine whether the new version actually improves.
View the complete static diagram
Sensor → DMA/RingBuffer → Preprocess → Inference → Postprocess → Event/Actuator
Code or command example
idf.py build flash monitor
# On-device concerns: arena, tensor arena, input normalization, inference latency
ESP_LOGI(TAG, "latency=%d ms", elapsed_ms);Hands-on lab
Lab notes and export
Engineering pitfall
Knowledge check
Answer all three multiple-choice questions, then submit. Answers stay hidden until submission.
Discuss this chapter on GitHub
Sign in with GitHub to ask a question, share measurements, or compare implementations. Comments are stored in this course's GitHub Discussions.
The embedded comments need JavaScript. You can also open the GitHub discussion area directly: Open GitHub Discussions ↗
Further reading
Bridge to the next chapter
Next, resource budgeting and vision pipelines merge across Flash, SRAM, PSRAM, DMA, preprocessing, and stable event postprocessing.