Attention Mechanisms
Turn sequence dependencies into an explicit, parallelizable dataflow
Suggested reading: about 20 min
Learning goal
Chapter keywords
| Keyword | Explanation | ESP32 engineering analogy |
|---|---|---|
| Attention | Information exchange that matches Query against Key and uses the scores to aggregate Value. | Like one query searching several indexed DMA descriptors at once. |
| Multi-head | Attention projected into several subspaces, run in parallel, and concatenated at the output. | Like parallel filters observing different protocol features. |
| Causal mask | An upper-triangular visibility constraint that prevents reading future tokens. | Like a receiver state machine that can read only bytes already received. |
| Positional encoding | Information about token order or relative distance injected into a non-recurrent attention model. | Like attaching a monotonic sequence number to each ring-buffer element. |
Bridge from the previous chapter
The previous chapter established the complete Transformer architecture and located Attention as the block sublayer responsible for cross-position communication. This chapter opens that large concept on its own and traces how Q, K, V, masks, heads, and positions determine aggregation.
Historical development
Attention began as an answer to fixed-vector compression in encoder-decoder models, became the Transformer dataflow, and later expanded through bidirectional pretraining, other modalities, and I/O-aware kernels. Structure and systems optimization jointly determine practical context length.
Learned alignment relieves fixed-length encoding pressure
Bahdanau and collaborators let each decoder step calculate alignment weights over encoder states instead of forcing one vector to carry an entire sentence.
Original neural-translation Attention paper ↗The Transformer replaces recurrence with multi-head Attention
Scaled dot products, multi-head self-attention, positional encodings, and causal masks were organized into an encoder-decoder that could train in parallel.
Original Transformer paper ↗BERT builds general language representations with bidirectional self-attention
Masked language modeling used both left and right context, demonstrating that pretrained Transformer encoders transfer across varied understanding tasks.
Original BERT paper ↗Transformer-XL adds segment recurrence and relative position
Reusing hidden state across segments and adopting relative positions let language models capture dependencies beyond fixed training fragments.
Original Transformer-XL paper ↗Vision Transformer carries a pure Transformer backbone into images
Images were divided into patch sequences and processed directly by a Transformer, showing that attention-based position interaction was not limited to natural language.
Original Vision Transformer paper ↗FlashAttention locates the bottleneck in HBM data movement
I/O-aware tiling reduced HBM traffic without approximating attention outputs, proving that the same formula can behave very differently under another memory schedule.
Original FlashAttention paper ↗Illustrated analogy
A parallel, multi-channel retrieval switching matrix
Every input port emits a query. Several matching circuits score visible ports using different clues, then retrieve data from corresponding channels. The channel outputs join before local processing. The matrix can serve a complete input batch at once, but it cannot process a next input that has not yet been generated.
Where the analogy stops: Heads are not necessarily human-nameable grammar or fact modules, and attention scores alone are not causal explanations. The analogy describes dataflow and parallel structure only.
Chapter walkthrough
The paper's first principle: shorten serial dependencies
The paper frames three design questions: how much computation each layer needs, how many steps must remain sequential, and how long the path is between distant positions. The Transformer keeps an encoder–decoder skeleton but replaces recurrence and convolution with multi-head self-attention and position-wise feed-forward networks, then adds residual connections, LayerNorm, and positional encoding for stable training and order. The whole input can therefore be processed with high parallelism and shorter long-range paths. That does not make generation parallel: the decoder still emits the next token autoregressively from the generated prefix.
Read scaling, heads, and causal masking from the formula
The core formula is Attention(Q,K,V)=softmax(QKᵀ/√d_k)V. As d_k grows, the dot-product variance grows too; feeding large values directly into softmax can saturate it and shrink gradients, so division by √d_k stabilizes the scores. Multi-head attention uses different learned projections to enter several representation subspaces, computes attention in parallel, concatenates the results, and projects them again. A causal mask makes future positions unavailable. Verify all three behaviors with four-token toy matrices before discussing kernels or caches.
Once recurrence is gone, order must enter explicitly
Attention can see a set of positions at once but does not know their order by itself. The paper therefore adds sinusoidal positional encodings to the embeddings and compares them with learned positional embeddings. This choice is part of the model contract: tokenizer, position encoding, causal mask, and cache positions must agree. Edge engineering cannot copy only the weight file; any change to these interfaces needs golden-vector checks to prove that outputs remain consistent.
Carry the paper's parallelism into prefill and decode
The paper's O(1) sequential-operation entry describes one self-attention layer processing a complete sequence; it does not mean autoregressive output has no ordering. Prefill handles an existing prompt at once and can parallelize across tokens. Decode adds one new token per round, must wait for the previous sampled result, and reads historical K/V. Modern runtimes therefore measure one throughput-oriented prefill separately from many latency-sensitive decode rounds; the paper's parallelism must not be used to hide decode's bandwidth bottleneck.
A Transformer block contains more than Attention
Attention exchanges information among positions, while a position-wise feed-forward network applies the same nonlinear transform at every position. Residual connections preserve short paths through a deep model, and LayerNorm stabilizes numerical scales. Engineering analysis should budget QKV projection, score computation, value aggregation, output projection, and the FFN separately. FFN weights often hold a large parameter share, while long contexts make attention intermediates and K/V state major memory and bandwidth consumers.
Self-attention gains short paths and parallelism, not free computation
The paper compares layers by complexity, required sequential operations, and maximum path length between positions. Self-attention shortens distant-dependency paths and processes training positions in parallel, but builds a correlation matrix that grows quadratically with sequence length. Long contexts therefore motivate sparse patterns, tiling, windows, or I/O-aware kernels. Those techniques alter computation, visibility, or data movement in distinct ways and should not be grouped as one generic acceleration.
Check masks, positions, and numerical stability from formula to implementation
Implementation bugs hide in broadcast mask direction, interactions between padding and causal masks, position offsets, softmax dtypes, and multi-head reshape order. Create tiny golden matrices: freeze Q/K/V and save raw scores, scaled scores, masked scores, softmax weights, and final outputs. Verify that every row sums to one and future weights are zero. This baseline exposes the first semantic break when kernels, quantization, or cache management are optimized later.
Interactive process
How one token reads context through multi-head self-attention
The player exposes projection, scoring, masking, normalization, aggregation, and block output instead of hiding Attention behind one box.
Embed positions · X = token embedding + position
Create an ordered representation at every position
Attention does not create order by itself
Loop / return condition: Every Transformer block repeats this path. Training positions can run in parallel, but a new generated position still waits for the previous token selection.
View the complete static diagram
token ids → embedding + position
│
Q = XWq, K = XWk, V = XWv
│
scores = QKᵀ/√dₖ + mask → softmax → weighted V
│
residual + norm → FFN → residual + normCode or command example
scores = Q @ K.T / sqrt(d_k)
scores = scores + causal_mask
weights = softmax(scores, axis=-1)
context = weights @ V
output = concat(heads) @ W_oHands-on lab
Lab notes and export
Engineering pitfall
Knowledge check
Answer all three multiple-choice questions, then submit. Answers stay hidden until submission.
Discuss this chapter on GitHub
Sign in with GitHub to ask a question, share measurements, or compare implementations. Comments are stored in this course's GitHub Discussions.
The embedded comments need JavaScript. You can also open the GitHub discussion area directly: Open GitHub Discussions ↗
Further reading
Bridge to the next chapter
Next comes KV Cache alone: why historical K/V is reusable, how many bytes every token adds, and how GQA, paging, and quantization alter capacity and bandwidth.
Next: Day 11 · KV Cache