Edge Vision Pipelines
Turn camera frames into stable events under memory, bandwidth, and real-time constraints
Suggested reading: about 20 min
Learning goal
Chapter keywords
| Keyword | Explanation | ESP32 engineering analogy |
|---|---|---|
| Peak live memory | The total size of all buffers that must be retained at the same instant. | Like task stacks and DMA descriptors that coexist during a context switch. |
| PSRAM | High-capacity RAM connected through an external interface. | Like an expansion warehouse with more space but different access and DMA rules. |
| Stride | The memory distance between adjacent pixel data on the same row. | Like the actual step of a DMA row transfer, which is not always equal to the visible width. |
| Letterbox | A resize method that preserves aspect ratio and pads the borders. | Like placing payloads of different lengths into fixed-size frames while preserving their proportions. |
Bridge from the previous chapter
Day 5 deployed a non-LLM model to an MCU. This chapter merges the resource ledger with the camera data path to test whether the system remains stable at real frame rates, memory pressure, and temperature.
Historical development
Edge vision grew through lightweight networks, memory hierarchy, DMA, and complete preprocessing and postprocessing. Peak compute matters only when data arrives on time, buffer lifetimes are correct, and outputs map back to the source.
Roofline plotted compute and bandwidth ceilings together
The Roofline model connected peak compute with sustainable memory bandwidth through arithmetic intensity, turning “optimize arithmetic or reduce movement?” into a measurable, discussable question.
Original UC Berkeley Roofline report ↗Eyeriss quantified the energy cost of moving data
Eyeriss introduced a row-stationary dataflow that reused weights, activations, and partial sums through local storage and inter-PE communication, emphasizing that moving less data is itself acceleration.
Original Eyeriss paper page ↗MCUNet co-designed networks and inference engines
MCUNet used TinyNAS to search for networks under device constraints and TinyEngine to schedule memory over whole-graph lifetimes, advancing joint optimization of models, runtimes, and hardware.
Original MCUNet paper ↗Deep CNNs and GPU training expand visual capability
AlexNet demonstrated striking ImageNet results with a deep convolutional network, accelerating the growth of vision backbones while confronting deployment teams with greater compute and memory pressure.
Original AlexNet paper ↗YOLO unifies detection into one network pass
YOLO predicts bounding boxes and class probabilities directly from an entire image, moving detection from multi-stage proposal pipelines toward a single-stage real-time flow and highlighting the importance of end-to-end latency.
Original YOLO paper ↗Vision models become part of complete device pipelines
Official projects such as ESP-WHO combine camera drivers, image processing, inference models, and sample applications, expanding deployment concerns from the network alone to the complete end-to-end path.
Official ESP-WHO repository ↗Illustrated analogy
A small restaurant during the dinner rush
A well-stocked cold room does not mean the kitchen can prepare many tables at once. Counter space determines how many plates can be spread out, a narrow aisle limits how quickly servers move food, and a very fast specialty oven accepts only the right trays. Every change of plate and every trip in or out costs time.
Where the analogy stops: The restaurant analogy is useful for capacity, throughput, and repacking costs, but real storage levels are not mutually exclusive rooms: cache replaces entries automatically, and DMA and CPU may access the same physical memory concurrently. Whether flush, invalidate, or alignment is required must come from the chip manual and measurement.
Chapter walkthrough
Peak memory comes from overlapping lifetimes, not adding file sizes
For every tensor in the computation graph, list its producer, last consumer, size, and memory capabilities, then draw live intervals before calculating the peak. Weights can often remain in mapped Flash, while activations, accumulators, and kernel workspace coexist in the same layer. Camera double buffers, task stacks, and network packets must also enter the system peak. A planner can reuse regions with non-overlapping lifetimes, but dynamic shapes, branches, and fallback alter the plan, so validate static estimates against the runtime high-water mark.
Use arithmetic intensity to decide whether the MAC unit or memory is waiting
Roughly divide each operator’s MACs by bytes moved from slower levels to build intuition for “operations per byte.” Low-intensity paths are more likely bandwidth-bound. A higher clock or more NPU units helps directly only when compute is the limit; under a bandwidth limit, first fuse operators, reuse tiled data, prevent intermediates from materializing, and improve layouts. Theoretical bandwidth is a ceiling. Sequential access, cache misses, bus contention, and refresh protocols determine the sustained value, so benchmark a realistic access pattern.
Zero-copy is an ownership protocol, not a pointer cast
Having camera DMA write directly into model input can remove one copy, but only when address capabilities, alignment, stride, cache coherence, and lifetime satisfy both parties. The CPU must not read before DMA completion, and inference must release the buffer before DMA reuses it; cached systems also need synchronization in the correct direction. Use an explicit state machine such as FREE, FILLING, READY, and IN_USE, and log timeouts and dropped frames. “Zero-copy” without these constraints often becomes intermittent tearing or stale data.
Choose output granularity from the product requirement first
If the only requirement is to determine whether a flame is present and the target occupies a stable ROI, begin with classification. Use detection when position and count matter, and consider segmentation when pixel-level area, boundaries, or traversable regions are required. Finer outputs usually increase labeling cost, post-processing, and memory. Also specify the minimum target size in pixels, acceptable occlusion, and camera distance, because input resolution sets an upper bound on available information. Compare candidates by replaying real product events, not by ranking a single metric measured on different datasets.
Treat a frame buffer as two-dimensional storage with a descriptor
A frame is more than width×height: it also has a format such as RGB565, YUV, or JPEG, a per-row stride, plane layout, alignment, and a valid region. JPEG must be decoded first, and YUV-to-RGB conversion must use the correct matrix and range. Treating padded rows as tightly packed pixels produces diagonal artifacts. Camera-driver buffers are usually pool-managed and must not be returned or overwritten before inference is complete. Verify the read path with color bars, checkerboards, and pixel probes before introducing the model; this isolates format errors quickly.
Keep a reversible ledger for every geometric transform
Pre-processing should record the original dimensions, ROI origin, scale factor, padding, and model input size. Stretch resizing changes object shape; letterboxing preserves aspect ratio but adds border padding, and either choice must match training. To map a detection box or segmentation mask back to the source image, reverse the transforms in order: remove padding, divide by the scale, add the ROI offset, and clip to valid bounds. Unit tests using corner markers and known rectangles reveal half-pixel and rounding errors more reliably than inspecting a few boxes by eye.
Validate the real-time pipeline through staged replay
Save a legally shareable set of source frames and retain the model input, raw output, NMS result, and remapped result for each frame as golden records. Compute per-stage summaries on the board and compare them with the PC implementation, while timing acquisition, pre-processing, inference, post-processing, and queuing separately. Stress the pipeline by varying exposure, frame rate, and consumer speed; verify which frame is dropped when the queue fills and that every buffer is eventually returned. Final FPS depends on the slowest stage and pipeline concurrency, not merely the reciprocal of one inference call.
Interactive process
From one image frame to one trustworthy event
The player preserves every pixel and coordinate transformation, making it easy to see how candidates are filtered and stabilized across frames.
Capture via DMA · YUV/RGB/JPEG + stride + timestamp
The camera fills a pooled buffer and transfers ownership
Every frame must carry a format descriptor
Loop / return condition: The next frame returns to DMA capture. Tracking state persists across frames, while the pixel buffer is returned to the camera pool immediately after publishing the result; these lifetimes must never be confused.
View the complete static diagram
camera DMA → frame buffer → ROI/resize/letterbox → model partition
▲ ownership/stride SRAM↔PSRAM↔NPU │
└──────── release ◄── NMS + temporal policy ◄──┘Code or command example
frame = camera_acquire()
input, transform = preprocess(frame)
raw = infer(input)
event = postprocess(raw, transform)
camera_release(frame)
assert peak_live_bytes <= memory_budgetHands-on lab
Lab notes and export
Engineering pitfall
Knowledge check
Answer all three multiple-choice questions, then submit. Answers stay hidden until submission.
Discuss this chapter on GitHub
Sign in with GitHub to ask a question, share measurements, or compare implementations. Comments are stored in this course's GitHub Discussions.
The embedded comments need JavaScript. You can also open the GitHub discussion area directly: Open GitHub Discussions ↗
Further reading
Bridge to the next chapter
The next chapter is the course boundary: leave conventional discriminative edge models and enter LLMs through their history, essential terms, and capability limits.
Next: Day 7 · LLM Foundations