Transformer Architecture
Understand the whole model before opening its Attention dataflow
Suggested reading: about 20 min
Learning goal
Chapter keywords
| Keyword | Explanation | ESP32 engineering analogy |
|---|---|---|
| Transformer block | A repeated sequence unit composed of sublayers such as Attention, an FFN, residual paths, and normalization. | Like a repeated SoC compute tile containing both an interconnect and a local processing path. |
| Decoder-only | A stack of decoder-style blocks with causal visibility, trained to generate the next token from one unified sequence. | Like a state machine that advances only from the protocol prefix already received. |
| FFN / MLP | A nonlinear feed-forward sublayer applied independently at each sequence position with shared parameters. | Like every packet passing through the same local field-transform unit. |
| Residual + Norm | Short paths that preserve representations together with controls on numerical scale through deep stacks. | Like a main-data bypass and level calibration keeping a long pipeline stable. |
Bridge from the previous chapter
Day 7 established the LLM history and vocabulary map, and Day 8 fixed the text-input ABI. This chapter treats the Transformer as its own architectural concept and traces the whole path from tokens and blocks to logits.
Historical development
Transformer history is not the expansion of one Attention formula. Its architectural skeleton, pretraining objectives, positional mechanisms, and scaling methods evolved together. Different branches retain related blocks while choosing distinct information flows for understanding, generation, and long-context tasks.
The Transformer establishes a recurrence-free encoder–decoder skeleton
The original paper assembled multi-head Attention, position-wise FFNs, residuals, LayerNorm, and positional encoding into stackable blocks, allowing high parallelism across sequence positions during training.
Original Transformer paper ↗GPT validates transfer after generative decoder pretraining
Generative pretraining applied a multilayer Transformer decoder to general language modeling before task fine-tuning, establishing an important starting point for the decoder-only route.
Original GPT paper ↗BERT strengthens bidirectional encoder pretraining
BERT used masked language modeling to incorporate context from both directions, demonstrating that an encoder-only Transformer could produce transferable language representations.
Original BERT paper ↗Transformer-XL extends dependencies across segments
Segment-level hidden-state recurrence and relative positional encoding preserved longer history beyond fixed training segments while making the boundaries of state reuse explicit.
Original Transformer-XL paper ↗GPT-3 demonstrates in-context learning at decoder-only scale
GPT-3 scaled an autoregressive Transformer and completed tasks from descriptions and examples in its prompt, bringing decoder-only scaling and in-context learning into the main line.
Original GPT-3 paper ↗Switch Transformer explores sparse expert FFNs
Sparse routing activated only selected expert FFNs for each token, increasing conditional parameter capacity while adding routing, load-balancing, and communication costs to the architecture.
Original Switch Transformer paper ↗Illustrated analogy
Read the SoC block diagram before opening the interconnect
Input indices pass through an interface and address marker, then cross repeated compute tiles. Each tile contains an interconnect for exchanging information, local nonlinear processing, bypasses, and level calibration. The output unit maps internal state to vocabulary scores. Studying only the interconnect would miss most weights, execution order, and compatibility contracts.
Where the analogy stops: A Transformer is not a physical SoC, and layers need not map to separate hardware units. The analogy only separates whole-model architecture, repeated blocks, cross-position exchange, and local computation.
Chapter walkthrough
Treat the Transformer as an architecture, not one formula
A Transformer is a framework for organizing sequence computation. Tokens enter embeddings and a positional mechanism, pass through a stack of repeated blocks, and reach final normalization and an output head that produces logits. A block usually contains Attention for cross-position exchange, an FFN for position-wise transformation, and residual and normalization paths that preserve deep signals. Fixing those boundaries first prevents one Attention formula from standing in for the whole model.
Encoders, decoders, and decoder-only models carry different information flows
An encoder lets every position read the full input and is suited to contextual representations. The original decoder combines causal self-attention over its generated prefix with cross-attention to encoder output. A decoder-only model places instructions, context, and answers in one causal sequence and applies one next-token objective. The families share the block idea but differ in masks, input contracts, and objectives, so the Transformer label alone does not make them interchangeable.
Embeddings, positions, and the output head define both ends of the sequence
A token ID is only a discrete index; an input embedding maps it into a hidden vector, and a positional encoding or rotary mechanism makes order and distance available. After the layer stack, hidden states pass through a final normalization and vocabulary projection to become logits. Some models tie that projection to the input embedding. Vocabulary size, hidden width, position parameters, and weight tying all change file size, operator shapes, and compatibility, so they belong in the model-package contract.
The FFN is the main nonlinear path at each position
Attention mixes information across positions, while an FFN independently applies the same expansion, activation, and contraction parameters at every position. The classic Transformer used a two-layer feed-forward network; modern LLMs often use gated MLP variants. Intermediate width and activation choice materially affect parameter count, memory traffic, and kernel coverage. Account for Attention projections and FFN weights separately instead of attributing all block cost to Attention.
Residuals and normalization govern stable deep propagation
Residual connections give each sublayer a short path that preserves and revises the existing representation, while LayerNorm or RMSNorm controls numerical scale. Models may use pre-norm, post-norm, parallel residuals, or different normalization formulas, and conversion cannot reorder them casually. Save golden tensors at block boundaries when validating a backend so the first error can be assigned to normalization, residual addition, or the sublayer computation.
Training parallelism and generation order live on different axes
During training the complete target sequence is known, so a causal mask can block future information while positions in one layer are computed as a batch; layers still follow graph dependencies. During generation, the next token does not exist until current logits have been sampled. The Transformer shortens paths among positions and improves training parallelism, but does not remove autoregressive time dependence. That distinction motivates the next Attention and KV-cache chapters.
Interactive process
How a token sequence crosses a complete Transformer
The player shows the model-level path and intentionally leaves Attention as one sublayer to be opened in the next chapter.
Receive token IDs · [batch, sequence]
Read the tokenizer's discrete indices
Vocabulary and embedding weights must match
Loop / return condition: Attention, residual, normalization, and FFN paths repeat for N configured layers. After sampling, each new autoregressive position crosses the complete stack again.
View the complete static diagram
token ids → embedding + position → N × Transformer block → final norm → logits
│
Attention → residual → FFN → residualCode or command example
x = token_embedding(token_ids) + position(position_ids)
for block in transformer_blocks:
x = block(x, mask=causal_mask)
logits = output_head(final_norm(x))Hands-on lab
Lab notes and export
Engineering pitfall
Knowledge check
Answer all three multiple-choice questions, then submit. Answers stay hidden until submission.
Discuss this chapter on GitHub
Sign in with GitHub to ask a question, share measurements, or compare implementations. Comments are stored in this course's GitHub Discussions.
The embedded comments need JavaScript. You can also open the GitHub discussion area directly: Open GitHub Discussions ↗
Further reading
Bridge to the next chapter
Day 10 gives Attention its own chapter, moving from learned alignment, Q/K/V, and causal masks to multi-head computation, quadratic cost, and I/O-aware implementations.