Operators and Kernels
Move from mathematical definitions to layouts, fusion, and hardware execution
Suggested reading: about 18 min
Learning goal
Chapter keywords
| Keyword | Explanation | ESP32 engineering analogy |
|---|---|---|
| Kernel | Low-level code that implements an operator for specific hardware. | Like a chip-specific driver implementing a shared peripheral API. |
| Layout | The arrangement of tensor dimensions in memory, such as NCHW or NHWC. | Like structure-field ordering and alignment rules. |
| Fusion | Combining several consecutive operators into one execution. | Like merging several small DMA transactions to reduce interrupts and traffic. |
| Tiling | Partitioning a large computation to fit registers or cache. | Like processing a long data stream with block-sized buffers. |
Bridge from the previous chapter
The previous chapter mapped floating-point values into low-bit representations. This chapter follows those values through operators, layouts, and actual kernels.
Historical development
Operator history intertwines two paths: inventing network structures that can express a task, and turning those mathematical structures into efficient kernels. After convolutional networks progressed from hierarchical receptive fields to end-to-end training, GPU primitive libraries separated optimized implementations from frameworks, and compilers automated fusion, layout, and scheduling. A modern runtime sits precisely between the mathematical graph and heterogeneous hardware.
The Neocognitron demonstrated hierarchical local receptive fields and shift robustness
Fukushima’s Neocognitron processed visual patterns hierarchically and pursued recognition under position changes. It was not a modern backpropagation-trained Conv kernel, but it brought local connectivity, feature hierarchies, and spatial reuse into the convolutional-network lineage.
Original Neocognitron paper ↗LeNet joined convolution, subsampling, and gradient training in an application system
LeCun and colleagues’ document-recognition work systematically demonstrated convolutional networks and end-to-end gradient learning. Operators were no longer isolated formulas; together with input geometry, weight sharing, and task postprocessing, they formed an executable pipeline.
Original LeNet document-recognition paper ↗cuDNN made deep-learning primitives a reusable high-performance library
cuDNN provided optimized GPU primitives such as convolution so frameworks did not need to rewrite every kernel for each generation of parallel hardware. Its paper compared deep-learning operator libraries with BLAS and showed how algorithm selection, memory use, and hardware optimization could hide behind stable interfaces.
Original cuDNN paper ↗TVM unified graph fusion and low-level scheduling in compilation search
TVM handled high-level operator fusion, hardware-intrinsic mapping, and memory-latency hiding while using cost models to search low-level optimizations. Kernel selection expanded from vendor-written libraries into a schedule space generated for CPUs, GPUs, FPGAs, and accelerators.
Original TVM OSDI paper ↗Execution Providers partitioned one graph among heterogeneous backends by capability
ONNX Runtime connects hardware implementations through Execution Providers and assigns supported nodes or subgraphs to each backend. Modern deployment therefore asks not only whether an NPU exists, but which nodes it claims, how much data crosses boundaries, and which paths fall back to the CPU.
Official ONNX Runtime Execution Providers documentation ↗Illustrated analogy
How one recipe is executed in a crowded professional kitchen
Operators are recipe instructions such as dice, sauté, and reduce: they specify inputs, actions, and results. A kernel is a particular cook’s technique on a particular stove. Arranging ingredients in order of use is layout; dicing and marinating on the same board is fusion; dividing a huge batch into portions that fit the pan is tiling. The recipe stays the same, yet serving time can differ by several times.
Where the analogy stops: The kitchen analogy cannot represent parallel threads, vector instructions, or floating-point rounding, and may imply that fusion is always beneficial. Real fusion is constrained by quantization scales, branch reuse, and backend support; both numerical equivalence and performance gain must be verified on target hardware.
Chapter walkthrough
An operator contract includes attributes, boundaries, and numerical conventions
Conv is more than a summation formula. Its contract includes stride, padding, dilation, groups, weight-dimension order, and output-shape rules; Softmax must specify both its normalization axis and numerically stable evaluation. Two runtimes may claim support for the same named operator while covering different dtype or attribute combinations. Export a contract table from actual nodes during integration, and validate it with asymmetric shapes and boundary inputs.
One Conv can map to several algorithms with different workspace tradeoffs
Direct convolution, im2col+GEMM, Winograd, and hardware-specific paths suit different regions. im2col reuses mature matrix-multiplication kernels but can expand into a large buffer; Winograd can reduce some multiplications but is sensitive to dimensions, numerical precision, and transform overhead. Measure backend algorithm choices with real shapes, available workspace, and the actual quantization mode instead of comparing theoretical MACs alone.
Layout determines which values are adjacent—and how smoothly vector units are fed
NCHW and NHWC change adjacency among channel and spatial dimensions. Even when dimension names agree, non-contiguous strides may trigger an implicit copy. One kernel may prefer contiguous channels while another accelerator requires a blocked layout, causing boundary conversion to read and write the entire activation. Treat transpose, reorder, and memcpy as first-class timed nodes in a profile, not unexplained framework overhead.
Tiling aims to make reuse happen in the fastest storage level
After partitioning M, N, and K, GEMM loads small input and weight tiles into registers or cache, performs repeated MACs, and only then writes back. A tile that is too large overflows the working set; one that is too small increases loop and boundary overhead. SIMD also imposes alignment and needs a mask or scalar path for tail elements. Test real addresses, alignments, and DMA origins on ESP32, because the ideal contiguity of desktop arrays does not appear automatically.
Accept fusion and accelerator partitioning by counting fewer materializations
Fusing Conv, Bias, and ReLU can retain intermediates in registers or on-chip SRAM. Fusion may be restricted, however, if another node consumes an intermediate or neighboring quantization scales are incompatible. NPU subgraph boundaries likewise introduce synchronization, layout changes, and copies. An acceptance report should list nodes before and after fusion, partition boundaries, bytes moved, and end-to-end time, rather than substituting a single-kernel speedup for system benefit.
Interactive process
How Conv–BN–ReLU becomes one tiled execution
The animation descends from an abstract graph into memory: first verify operator contracts, then choose fusion and layout, and finally cycle tiles through load, compute, and write-back in on-chip buffers.
Expand operator contracts · Conv(stride,pad,groups) → BN → ReLU
Attach attribute cards to nodes and update input/output shapes together
Same-name operators are semantically compatible only when attributes, axes, and dtypes also match
Loop / return condition: Tiles iterate across the output space until completion. Changing layout, tile size, or backend resets counters while retaining the previous run’s bytes, workspace, and latency side by side.
View the complete static diagram
Graph: Conv → BN → ReLU Optimizer: Conv+BN+ReLU fusion Kernel: tile → load → MAC → store
Code or command example
for (m_tile : M)
for (n_tile : N)
acc = 0
for (k_tile : K) acc += A * B
C = acc + biasHands-on lab
Lab notes and export
Engineering pitfall
Knowledge check
Answer all three multiple-choice questions, then submit. Answers stay hidden until submission.
Discuss this chapter on GitHub
Sign in with GitHub to ask a question, share measurements, or compare implementations. Comments are stored in this course's GitHub Discussions.
The embedded comments need JavaScript. You can also open the GitHub discussion area directly: Open GitHub Discussions ↗
Further reading
Bridge to the next chapter
Next, data, training, conversion, quantization, and firmware runtime become a complete non-LLM edge deployment loop.