Neural Network Foundations
Build neural-network intuition from tensors, layers, and loss functions
Suggested reading: about 18 min
Learning goal
Chapter keywords
| Keyword | Explanation | ESP32 engineering analogy |
|---|---|---|
| Tensor | A multidimensional numeric block with a shape and data type—the common representation for model inputs, weights, and intermediate results. | Like a DMA buffer annotated with sample count, channel count, and bit width. |
| Shape | The length and meaning of each dimension, such as batch, channel, height, and width in NCHW. | Like the field layout of a protocol frame: the bytes remain when the order is wrong, but their meaning is corrupted. |
| Activation function | A nonlinearity applied after a linear transform that lets multilayer networks express complex boundaries. | Like a conditional branch in a state machine that cannot simply be folded away. |
| MAC | One multiply–accumulate operation, commonly used for a rough estimate of compute. | Like the number of iterations in an inner DSP loop, still subject to the cost of fetching data. |
Bridge from the previous chapter
This is the starting point of the course. You already know the ESP32 path in which a peripheral produces data, DMA moves it, and a task consumes it. This chapter abstracts that path into neural-network inputs, layers, and outputs, establishing a shared language for the training and deployment discussions ahead.
Historical development
Neural networks did not suddenly appear in the age of “large models.” They followed a long path from logical neurons and trainable linear classifiers, through credit assignment in multilayer networks, to large-scale representation learning on GPUs. The tensors, operators, automatic differentiation, and deployment graphs used in edge engineering today are layered interfaces accumulated along that path.
The neuron was first written as a computable logical unit
McCulloch and Pitts described neural activity with threshold units and connection networks, showing that simple units could be composed into logical behavior. Their work had no modern training algorithm, but it established the computational view that complex functions can emerge from many uniform, connected units.
Original paper by McCulloch and Pitts ↗The perceptron turned “connections” into weights corrected by examples
Rosenblatt’s perceptron introduced a mechanism for adjusting connection weights from classification errors, moving neural networks from hand-crafted logical structures toward data-driven learning. It mainly addressed linearly separable problems, but made “model parameters are the result of training” a central idea for later networks.
Original perceptron paper by Rosenblatt ↗Backpropagation gave hidden layers computable error signals
Rumelhart, Hinton, and Williams demonstrated how error backpropagation calculates derivatives layer by layer and adjusts weights. Hidden units no longer needed hand-assigned meanings; they could form internal representations for a task, giving multilayer nonlinear networks a practical, unified training method.
Original Nature backpropagation paper ↗AlexNet carried deep representation learning into the big-data and GPU era
AlexNet trained a deep convolutional network on a large image dataset and used GPUs for dense tensor computation. The breakthrough came not only from its architecture, but from data, parallel computing, and regularization arriving together. Parameter count, MACs, memory, and throughput consequently became parts of one engineering budget.
Original AlexNet paper ↗Imperative tensor programs and automatic differentiation became everyday tools
The PyTorch paper summarized a combination of an imperative front end, dynamic graphs, and a high-performance tensor backend. Researchers could write control flow like ordinary programs while the system recorded operations and calculated gradients. Today’s need to confirm dynamic paths before export follows directly from that flexibility.
Original PyTorch systems paper ↗Illustrated analogy
A bank of mixing consoles that tunes itself from sound-check notes
Imagine a band’s multitrack recording entering several rows of mixing consoles. Each row mixes channels according to knob positions, then passes them through noise gates that admit only certain signals. At the end, an engineer compares the mix with a reference recording and sends correction notes—“too loud here, too weak there”—backward along the signal path. After repeated sound checks, the knob positions are the learned weights.
Where the analogy stops: This analogy only helps explain signal transformation and error feedback. A real network “knob” participates in high-dimensional batched operations, and a gradient is a precise composition of local derivatives rather than a subjective opinion. The analogy also cannot represent parameter sharing, convolution layouts, or coupling introduced by numerical precision.
Chapter walkthrough
Read strides and axis semantics, not just shape
The same 3,072 numbers mean entirely different things in [1,3,32,32] and [1,32,32,3]; even with the same shape, a transposed view may be non-contiguous. Layers normalize, convolve, or reduce along agreed axes, while broadcasting can make a wrong shape “run successfully.” When reading a network, label every axis with N/C/H/W, units, and dtype, and print strides at boundaries so that executability is never mistaken for semantic correctness.
Nonlinearity changes the decision boundaries that can be composed
An affine layer only rotates, scales, and translates; multiple affine maps can still be folded into one matrix multiplication. Activations such as ReLU divide the input space into regions governed by different linear relationships, so stacking layers genuinely increases expressive power. In engineering work, also inspect activation ranges: large dead regions may signal shifted inputs or poor initialization, while excessively large values may saturate during low-precision deployment.
The loss is the objective interface; gradients are the layer-by-layer credit ledger
A loss function specifies which discrepancies deserve a penalty; the chain rule then propagates the sensitivity of total error to each intermediate value, one layer at a time. A zero gradient does not necessarily mean learning is complete—it may come from saturated activations, a detached computation graph, or numerical underflow. An exploding gradient can likewise make one update overshoot the useful region. Training experiments should record loss, gradient norms, and parameter-update magnitudes rather than watching final accuracy alone.
Parameters, MACs, and peak activations answer three different questions
Parameter count roughly estimates weight storage; MAC count roughly estimates multiply–accumulate work; activation lifetimes determine how large the runtime workbench must be. A convolution may have few weights yet produce a huge feature map, while an elementwise operator has almost no parameters but still reads and writes an entire tensor. For an edge-device budget, draw live intervals in execution order and add workspace, alignment padding, and I/O buffers to the peak.
Use finite differences and intermediate tensors to verify that the math is right
For a tiny network, freeze weights and inputs and save each layer output, the loss, and analytical gradients. Then perturb one parameter by a small positive and negative ε and approximate its derivative from the two loss values. A mismatch often comes from a transpose, batch reduction, in-place mutation, or an activation boundary. Deployment does not need backpropagation, but these golden tensors remain valuable for checking that export, quantization, and kernels preserve forward semantics.
Interactive process
One training pulse: how data becomes a weight update
Each step in the player illuminates a tensor and its owner. Signals flow to the right in the first half, gradients travel back to the left in the second, and the weight scale moves slightly at the end.
Load samples · x:[B,2], y:[B,1]
Highlight the batch axis and send the input into the first layer
Shape, dtype, and axis semantics must be fixed before later computations can share a contract
Loop / return condition: The updated weights receive the next batch. The animation returns to “Load samples” while preserving a gradually descending loss trace as state across iterations.
View the complete static diagram
Input x │ Linear(Wx+b) ▼ Activation f(·) ──► Prediction ŷ ──► Loss(ŷ,y) ▲ │ └──────── Gradient backprop ◄──────────┘
Code or command example
y = x @ W.T + b
y = relu(y)
loss = mean((y - target) ** 2)
loss.backward()Hands-on lab
Lab notes and export
Engineering pitfall
Knowledge check
Answer all three multiple-choice questions, then submit. Answers stay hidden until submission.
Discuss this chapter on GitHub
Sign in with GitHub to ask a question, share measurements, or compare implementations. Comments are stored in this course's GitHub Discussions.
The embedded comments need JavaScript. You can also open the GitHub discussion area directly: Open GitHub Discussions ↗
Further reading
Bridge to the next chapter
Next, training mode, inference mode, computation graphs, weights, and format contracts become one verifiable model-delivery chain.