LLM Runtime
Organize model packages, generation loops, and heterogeneous backends into an observable execution path
Suggested reading: about 20 min
Learning goal
Chapter keywords
| Keyword | Explanation | ESP32 engineering analogy |
|---|---|---|
| Context | The tokens and KV state held by one inference session. | Like the protocol-state block for one connection. |
| Cancellation | A mechanism that aborts an obsolete request and releases its resources. | Like reclaiming DMA and task resources after a socket closes. |
| Backend | The implementation layer that maps runtime requests onto specific hardware. | Like a concrete peripheral driver behind a common driver interface. |
| Offload | Assigning some layers or subgraphs to an accelerator. | Like handing a computation to a coprocessor. |
Bridge from the previous chapter
The first six days established the conventional edge-model execution chain, and Days 7–12 added LLM inputs, architecture, Attention, KV, and model packages. This chapter places those static contracts inside a real runtime so request scheduling and heterogeneous backends become one traceable data and control path.
Historical development
LLM runtimes and on-device frameworks jointly connect model semantics to hardware reality. One manages generation state and request lifecycles; the other maps operators to heterogeneous devices through graph transforms, compilation, and delegates. Their histories meet in an execution path that must be observable and reversible.
The Transformer establishes parallel prefill and autoregressive decode
Attention Is All You Need replaces recurrence with an attention-only architecture, allowing an entire input to be processed with high parallelism while generation still emits the next token from the accumulated prefix. Modern runtimes therefore divide naturally into one throughput-oriented prefill and many latency-sensitive decode rounds.
Original Transformer paper ↗llama.cpp advances local LLM execution on general-purpose hardware
With a lightweight C/C++ implementation, llama.cpp brings model loading, quantized kernels, mixed CPU/GPU execution, and the command-line generation loop into one project. Local inference moves from research script to embeddable process, and mmap, thread count, context, and backend become ordinary deployment parameters.
Official llama.cpp repository ↗Local runtimes acquire complete service semantics
Modern llama-server versions include parallel decoding, continuous batching, streaming APIs, monitoring, structured output, and cancellation in the service layer. Engineering emphasis moves from “generate one sentence” to resource isolation, observability, protocol compatibility, and the safety boundary around untrusted tool calls.
Official llama.cpp Server documentation ↗ONNX advances model exchange and operator-version contracts
ONNX describes a model with graphs, nodes, initializers, types, and opsets, allowing training frameworks and executors to collaborate around a common IR. It answers “what is expressed,” not whether the target chip has an efficient implementation. This separation later becomes the first yardstick for comparing runtimes.
Official ONNX concepts documentation ↗llama.cpp proves the value of a specialized lightweight edge runtime
For autoregressive LLMs, llama.cpp tightly integrates GGUF, quantized kernels, KV caching, and multiple CPU/GPU backends, covering desktops and edge devices with relatively few dependencies. It represents another path: deep vertical integration for a primary workload rather than support for arbitrary training graphs.
Official llama.cpp repository ↗Competition shifts to compiled artifacts, heterogeneous partitioning, and observability
Systems such as MLC LLM separate weight conversion from compiling a target model library and generate inference logic for WebGPU, mobile GPUs, or native platforms. Engineering comparisons shift from API style to which graphs are supported, when compilation happens, how much crosses partitions, whether fallback is traceable, and whether measurements remain reproducible after an upgrade.
Official MLC LLM compilation documentation ↗Illustrated analogy
Think of the runtime as a busy noodle shop
Model weights are the fixed recipes on the wall and can be loaded once for reuse. Each customer's prompt is a new order; prefill is the cook reading every customization at once and preparing the work surface, while the KV cache is that table's private tray of partially prepared ingredients. Decode then performs one small step and serves one token at a time. The sampler chooses among acceptable seasonings, and streaming lets the customer begin eating before the whole meal is finished. If the customer leaves, a cancellation bell must stop the kitchen immediately and return the tray.
Where the analogy stops: The noodle-shop analogy breaks down for parallel execution: a GPU batch is not several cooks each preparing one bowl, but one operation combining similar matrix work from different requests. A token is not an independent dish either; it changes every probability in the next step. The analogy explains lifecycle and ownership, not throughput, VRAM layout, or sampling mathematics.
Chapter walkthrough
Distinguish process resources from session state first
Weight mappings, backends, and thread pools usually belong to the process and can be shared across requests. Token sequences, KV caches, sampler state, and stopping conditions belong to a session and must be isolated. Draw separate lifecycles for both object classes: weights are released when the process exits, while a context is returned as soon as its request ends. Putting session pointers into a global singleton causes histories to overwrite each other under concurrency; reloading weights for every request lets initialization dominate TTFT.
The sampler is a reproducible decision pipeline
After the model emits logits, repetition penalties, temperature scaling, top-k, top-p, and random selection alter the candidate distribution in a defined order. A different order or random seed can make output diverge. For correctness debugging, begin with greedy decoding or a fixed seed and preserve sampling parameters plus summaries of the first few logits. Tool calling should also use grammar or schema constraints, but parseable structure does not prove that a command is authorized.
Streaming and cancellation must share one control path
SSE, WebSocket, and chunked HTTP are only output channels. The harder question is whether a decode task observes a cancellation flag at its next safe point after the client disconnects, then leaves the queue and releases KV resources. Give every request an explicit state machine: queued, prefill, decode, completed, cancelled, and failed, with each terminal state reclaiming resources exactly once. Inject slow consumers, partial packets, timeouts, and repeated cancellation, and check for hanging threads and memory leaks.
Use a four-layer model to unpack framework claims
Write down model format, runtime, backend, and kernel separately. GGUF or an exported graph expresses the model; the runtime manages lifecycle and scheduling; the backend connects CPU/GPU/NPU; kernels execute concrete layouts and dtypes. When a framework claims to “support” a chip, ask which model architectures, operators, and quantization types actually use specialized kernels. A version mismatch at any layer can cause load failure, silent fallback, or extra conversion.
A partition's value depends on boundaries, not node coverage
A delegate usually selects contiguous supported subgraphs. Boundaries may require device copies, layout transforms, quantization or dequantization, and synchronization. Even if 90% of nodes are assigned to an NPU, unsupported nodes interleaved through every layer can make transfers dominate latency. Record the partition count, input and output bytes for each partition, and execution time, then disable the delegate for a control run. Node coverage is a clue, not a performance conclusion.
AOT and JIT redistribute deployment responsibility
Ahead-of-time compilation completes lowering, fusion, and target-code generation before release. It can shrink the device runtime and reduce startup jitter, but requires per-chip and per-shape artifact management. JIT adapts to more dynamic situations on site but adds compilation latency, caches, and toolchain dependencies. Firmware and offline products often favor AOT, whereas desktop applications can tolerate JIT. In both cases, record compiler versions, target features, and generation settings in the artifact manifest.
Make framework selection a maintainable acceptance matrix
For every candidate, list target OS and chip, model architecture, maximum context, quantization formats, package size, cold start, peak memory, PP/TG, power, license, and debugging facilities. Prepare normal, boundary, and failing models, including an unsupported operator, a dynamic shape, and a low-memory case; observe whether errors and fallback are visible. Rerun the matrix after every upgrade. A consistent second-place framework whose regressions remain explainable is often a better product choice than a one-time benchmark winner that cannot be diagnosed.
Interactive process
How a model graph is partitioned across CPU, GPU, and NPU
The step-by-step player expands abstract “hardware acceleration” into capability queries, partitioning, lowering, boundary transfers, and fallback evidence.
Read the graph contract · ops + shape + dtype + layout + quant params
Validate model version and I/O, then establish a runnable CPU baseline
A readable format does not mean every node has a target-backend implementation
Loop / return condition: If a partition loses performance, return to “Form partitions” and try to enlarge a contiguous subgraph, remove layout conversion, or keep that region on CPU. If results disagree, return to “Read the graph contract” and compare every boundary. Change only one backend or compiler option at a time.
View the complete static diagram
model package → loader/compiler → runtime request loop
│
CPU baseline ↔ GPU delegate ↔ NPU partition
queue → prefill → decode → sample → stream / cancel / reclaimCode or command example
engine = load(model_package, backend_policy)
for request in scheduler:
state = prefill(request.tokens)
while not request.done:
logits, state = decode_one(state)
request.stream(sampler(logits))
reclaim(state) # cancellation and errors share this pathHands-on lab
Lab notes and export
Engineering pitfall
Knowledge check
Answer all three multiple-choice questions, then submit. Answers stay hidden until submission.
Discuss this chapter on GitHub
Sign in with GitHub to ask a question, share measurements, or compare implementations. Comments are stored in this course's GitHub Discussions.
The embedded comments need JavaScript. You can also open the GitHub discussion area directly: Open GitHub Discussions ↗
Further reading
Bridge to the next chapter
Next, single-machine benchmarks expand into chips, interconnects, parallel training, online serving, and SLOs as one cross-scale evidence chain.