LLM Foundations
Map the origins, historical path, mechanics, and essential terminology of language models
Suggested reading: about 20 min
Learning goal
Chapter keywords
| Keyword | Explanation | ESP32 engineering analogy |
|---|---|---|
| Language model | A model that estimates sequence probability or the conditional probability of the next token. | Like predicting the next protocol field's type and range from fields already received. |
| Parameter | A trained weight reused unchanged during inference—the model's long-lived statistical state. | Like lookup constants and control coefficients compiled into firmware and reused across requests. |
| Pretraining / alignment | Pretraining learns broad patterns; alignment uses instructions, preferences, or rules to shape usable behavior. | Like building a general parser, then constraining commands and error handling to a product specification. |
| Hallucination | Fluent model output that lacks factual support or conflicts with reality. | Like a checksum-valid, well-formed packet whose payload semantics are still wrong. |
Bridge from the previous chapter
The first six days established the conventional AI engineering loop from neural networks and model artifacts to constrained edge deployment and vision pipelines. Today crosses into generative language models by building a map that is not distorted by tool names or parameter counts.
Historical development
LLMs did not arrive in one paper. Probabilistic language modeling, distributed representations, parallel architectures, scaled pretraining, and human-preference alignment accumulated over decades. Their milestones reveal which capabilities come from objectives, structures, and deployment systems.
Information theory measures language uncertainty and predictability
Shannon used entropy and conditional probability to study communication and English sequences, establishing a starting point for treating language as a stochastic process whose uncertainty falls with context.
Original Shannon paper ↗A neural probabilistic language model learns distributed word representations
Bengio and collaborators estimated next-word probabilities with shared continuous word vectors and a neural network, letting related contexts share statistical strength and reducing the n-gram curse of dimensionality.
Original neural language-model paper ↗The Transformer rebuilds sequence modeling around Attention
Attention Is All You Need removed recurrent and convolutional backbones, organizing self-attention, feed-forward layers, residual paths, and positional encodings into a parallelizable encoder-decoder.
Original Transformer paper ↗BERT shows that broad pretraining can transfer to many understanding tasks
Bidirectional Transformer pretraining followed by task fine-tuning demonstrated a reusable base representation that could adapt to many natural-language understanding workloads.
Original BERT paper ↗GPT-3 demonstrates in-context learning in a scaled autoregressive model
GPT-3 performed varied tasks from instructions or a few prompt examples without parameter updates, making scale, prompt design, and in-context learning central research and product variables.
Original GPT-3 paper ↗InstructGPT adds instruction following and human preference to the alignment chain
Supervised demonstrations, a reward model, and reinforcement learning from human feedback were combined to make a base model more likely to follow user intent while exposing the importance of alignment evaluation and safety boundaries.
Original InstructGPT paper ↗Illustrated analogy
A repeatedly revised but read-only-on-site protocol predictor
A lab trains predictor firmware from a large archive of packets and compresses recurring structure into parameters. A product team then calibrates how it responds to commands. For a live request, the device loads fixed firmware, places the current packet in a session buffer, and predicts subsequent parts. The output may be smooth, but critical fields still pass deterministic validation.
Where the analogy stops: LLM parameters are not a database of individually queryable facts, and tokens are not fixed semantic protocol fields. The analogy distinguishes training-time state, request-time state, and external validation responsibility.
Chapter walkthrough
Start from the task definition: an LLM is a conditional probability model
A language model learns a probability distribution for the next token conditioned on the visible prefix. During training, shifting a real sequence by one position creates supervision at every position; during inference, one selected token is appended before predicting again. This objective pressures the model to compress syntax, semantics, factual co-occurrence, and task formats. It optimizes predictive likelihood, however—not database consistency, complete logical proof, or real-world safety.
From statistical counts to distributed representations, the recurring problem is generalization
Early n-gram models counted the next word after a finite history. They were simple and interpretable but suffered from unseen combinations and dimensional explosion. Neural language models mapped discrete words into continuous embeddings and shared parameters across similar contexts. Methods such as word2vec made the reuse of distributed representations especially visible. LLMs are not a sudden magic trick; they join decades of language objectives, representation learning, compute scaling, and data engineering.
Transformers made long dependencies and large-scale parallel training more practical
Recurrent networks pass state through a sequence one step at a time, creating long paths for both parallel execution and distant credit assignment. Transformers use self-attention for direct position-to-position interaction, then add position-wise feed-forward networks, residual paths, normalization, and positional mechanisms. Training can process a full sequence in parallel, but autoregressive output still has order. Input processing and output generation are distinct workloads explored later through Attention, Prefill, Decode, and KV cache.
Pretraining, fine-tuning, and alignment have different responsibilities
Pretraining learns general representations and generation behavior from broad corpora. Supervised fine-tuning shapes a narrower task and response format. Instruction alignment uses demonstrations, preferences, or rules to make behavior more useful and safer. All can change weights, but their data, objectives, and acceptance criteria differ. In-context learning is different again: examples in a prompt affect only the current context and do not permanently update parameters. Calling every process “training” obscures cost, privacy, rollback, and ownership.
Scale brings capabilities while amplifying data, evaluation, and system problems
Parameter count, training data, and compute jointly affect capacity, and no single number is a universal capability score. Larger models may show stronger few-shot transfer and task generalization, but demand more training resources, weight storage, inference bandwidth, and data governance. A foundation model is a pretrained base adaptable to many tasks, not a certification that it understands everything. Comparisons must fix the workload, prompt template, context, precision, hardware, and evaluation set.
Place common terms in the correct lifecycle
A token is a discrete identifier defined by a tokenizer; an embedding is its vector representation; parameters are long-lived trained state reused across requests; context is the token sequence visible to one request; KV cache is temporary per-layer state produced from that sequence; logits are unnormalized next-token scores; and sampling chooses the emitted token. Once these lifetimes are explicit, the model file, session memory, serving scheduler, and user experience can share one resource ledger.
Fluent probability is not factual reliability
A model continues text according to its training distribution and current context, so it can produce grammatical but false, fabricated, or contradictory claims. Retrieval, tools, structured constraints, and human review reduce risk without automatically turning a probabilistic model into a source of truth. An edge product must define where the model may advise, keep actuation, safety decisions, and irreversible actions behind deterministic checks, and continuously evaluate real failure examples.
Interactive process
How language capability travels from a corpus into one answer
The player separates long-lived training from short-lived request inference so parameters, context, and cache do not collapse into one vague state.
Curate the corpus · documents + provenance
Clean, deduplicate, filter, and record sources
Both capability and risk begin in the data distribution
Loop / return condition: Generation loops at “Generate token by token” until EOS, a limit, or cancellation; only offline training or fine-tuning returns to parameter updates.
View the complete static diagram
Text corpus ──tokenize──► token sequences ──pretrain: predict next token──► base model User instruction ──template──► context ──token-by-token inference──► distribution ──sample──► output
Code or command example
tokens = tokenizer(chat_template(messages))
for token in tokens:
state = model(token, state)
while not stop:
logits, state = model(next_token, state)
next_token = sample(logits)Hands-on lab
Lab notes and export
Engineering pitfall
Knowledge check
Answer all three multiple-choice questions, then submit. Answers stay hidden until submission.
Discuss this chapter on GitHub
Sign in with GitHub to ask a question, share measurements, or compare implementations. Comments are stored in this course's GitHub Discussions.
The embedded comments need JavaScript. You can also open the GitHub discussion area directly: Open GitHub Discussions ↗
Further reading
Bridge to the next chapter
Next, Tokenizer traces text into token IDs and treats the vocabulary, special tokens, and chat template as a model ABI.
Next: Day 8 · Tokenizer