KV Cache
Trade session memory for reuse of historical projections during Decode
Suggested reading: about 19 min
Learning goal
Chapter keywords
| Keyword | Explanation | ESP32 engineering analogy |
|---|---|---|
| Prefill | The stage that processes an existing prompt in parallel and establishes initial per-layer K/V state. | Like parsing a complete handshake packet when a connection begins. |
| Decode | The stage that appends one position and generates the next token after the previous output is known. | Like an event loop advancing only after the prior state result arrives. |
| KV cache | Session state containing Key and Value projections produced by historical tokens at every layer. | Like layered indexes and data descriptors retained for validated fields. |
| GQA / MQA | Structures in which several Query heads share fewer K/V heads, reducing cache width per token. | Like several consumers sharing fewer read-only index tables. |
Bridge from the previous chapter
The previous chapter exposed Attention's Q/K/V, causal masks, and multi-head aggregation as explicit dataflow. This chapter focuses only on reusing historical K/V during generation and the resulting session-memory, bandwidth, and reclamation responsibilities.
Historical development
The history of the KV cache reflects autoregressive generation's transition from mathematical feasibility to system scalability. The Transformer defined attention; MQA and GQA then reduced state per token, while paging, reuse, and quantization began addressing the memory-management problems of dynamic sessions.
The Transformer establishes scaled dot-product and multi-head attention
Attention Is All You Need frames the bottleneck as computation, sequential steps, and long-range path length, then replaces recurrence with an encoder–decoder built from scaled dot-product attention, multiple heads, positional encoding, and a causal mask. Input processing becomes easier to parallelize, while autoregressive output still emits one token from the prefix at a time.
Original Attention Is All You Need paper ↗MQA targets incremental-decoding bandwidth directly
Multi-Query Attention lets multiple query heads share one set of K/V heads, substantially shrinking the K/V tensors repeatedly read during incremental decoding.
Original MQA paper ↗GQA adds a step between quality and cache size
Grouped-Query Attention lets a group of query heads share one K/V head, providing a tunable compromise between conventional multi-head attention and single-group MQA.
Original GQA paper ↗PagedAttention addresses fragmentation from dynamic sessions
Borrowing from virtual-memory paging, PagedAttention maps the variably growing KV cache of each request onto non-contiguous blocks and supports more flexible sharing.
Original PagedAttention paper ↗KV quantization becomes an optimization area of its own
KIVI analyzes the numerical distributions of keys and values and quantizes them along different dimensions, showing that cache precision can be budgeted independently, just like weight precision.
Original KIVI paper ↗Local runtimes expose cache types and offload policies
Executors such as llama.cpp make K/V data types, offload, context length, and cache reuse configurable; the cache is now a deployment setting rather than a hidden implementation detail.
Official llama.cpp CLI documentation ↗Illustrated analogy
A detective's growing case-file index cards
After reading each page of testimony, the detective does not reread the case from page one. Instead, every layer of analysis produces two index cards: one says how to locate this clue in the future, and the other says what content to retrieve when it is found. A new question carries a query card across the historical index, forms a judgment, and appends the new testimony's cards.
Where the analogy stops: A KV cache is not a readable summary, fact database, or lossless copy of the source tokens. It contains intermediate tensors specific to model weights, positional encoding, and precision. Changing the model, moving prefix positions, or editing any history generally invalidates the old cards, and caching does not make attention itself constant-cost.
Chapter walkthrough
Use one layer of causal attention to see why K and V are stored
Linear projections turn each token's hidden state into Q, K, and V. The current Q is compared with all visible historical K values; after the causal mask and softmax, the resulting scores weight historical V values. With fixed model weights, an old token's K/V at that layer does not change when a new token arrives, so it can be cached. Q serves only the current computation and normally need not be retained. The cache avoids recomputing historical K/V projections, but attention still reads the historical cache and its cost grows with context length.
Calculate the cache slope from the model architecture first
A common approximation is batch×layers×2×tokens×kv_heads×head_dim×bytes per element, where 2 represents K and V; implementations add alignment and paging metadata. Do not substitute attention heads for kv_heads, because they differ in GQA/MQA models. Express the formula as “bytes added per new token per session,” then add weights, temporary workspace, and runtime overhead. If the measured slope differs, inspect the cache dtype, sliding-window policy, and preallocation.
Validate compression, capacity, and state together
MQA and GQA reduce KV heads, low-bit caches reduce capacity and bandwidth, sliding windows limit access to distant history, and paging primarily reduces fragmentation; these savings must not be conflated. Hold the model and sampling parameters fixed while increasing prompt length, and record prefill time, per-token latency, and KV bytes. For 32 layers, 8 KV heads, head_dim 128, and FP16, each token is about 128 KiB, so 4096 tokens are about 512 MiB before other overhead. Stress a paged cache with random start, growth, and cancellation, verify that free blocks return to baseline, and compare logits with a cache-disabled reference to catch position or eviction corruption.
Budget Prefill and Decode separately
Prefill processes all prompt tokens in a batch and usually has higher parallelism and compute density. Decode creates only one position per round while repeatedly reading growing historical K/V, so memory bandwidth and scheduling overhead often dominate. TTFT includes queuing, tokenization, prefill, and first-token sampling; ITL measures subsequent token intervals. One average tokens/s number conceals different bottlenecks for long prompts, short answers, and concurrent sessions.
Cache correctness depends on model, prefix, position, and precision
K/V tensors are produced by specific layer weights, a specific token prefix, positional semantics, and cache dtype. Changing the model or adapter, editing the middle of a prefix, shifting positions, or using incompatible rotary parameters normally invalidates old state. A prompt-cache key must cover those identities. Cancellation, timeout, and eviction must return every page or slot, or a long-running service develops a hidden capacity leak.
Interactive process
How a new token reads and extends the KV cache
The player separates the one-time prefill from repeated decode and shows the cache growing step by step.
Prefill the prompt · N prompt tokens
Process the existing sequence in parallel and generate K/V at every layer
The first pass concentrates computation and establishes historical state
Loop / return condition: “Append and repeat” returns to “Produce a new query” until EOS, a length limit, or cancellation. Cancellation must release the pages or slots owned by that sequence.
View the complete static diagram
Prompt tokens ──Prefill──► per-layer K/V cache
New token ──Q,K,V──► Q × cached Kᵀ → weighted V → logits → sample
└──── append new K/V and repeat ────┘Code or command example
kv_bytes = batch * layers * 2 * tokens * kv_heads * head_dim * bytes_per_value
bytes_per_new_token = batch * layers * 2 * kv_heads * head_dim * bytes_per_value
# Also budget alignment, page metadata, and preallocation policyHands-on lab
Lab notes and export
Engineering pitfall
Knowledge check
Answer all three multiple-choice questions, then submit. Answers stay hidden until submission.
Discuss this chapter on GitHub
Sign in with GitHub to ask a question, share measurements, or compare implementations. Comments are stored in this course's GitHub Discussions.
The embedded comments need JavaScript. You can also open the GitHub discussion area directly: Open GitHub Discussions ↗
Further reading
Bridge to the next chapter
Next comes LLM quantization and GGUF, connecting weights, tensor types, block overhead, metadata, and output quality in one verifiable model package.