Shipping Edge LLMs

Learning goal

Chapter keywords

Bridge from the previous chapter

The first fourteen days progressed from neural foundations to LLM structure, runtimes, quantization, frameworks, and cross-scale performance. This chapter compresses everything into a product: who has authority, where every byte and joule goes, how failure remains safe, and how versions ship and roll back.

Chapter walkthrough

Work backward from acceptance criteria to system boundaries

First specify required events, end-to-end latency, offline duration, false-positive and false-negative budgets, safe states, power, and cost. Then decide whether the model belongs on the MCU, an Edge Host, or the cloud. If an emergency stop must still occur in milliseconds without a network, neither the decision nor actuation can depend on a host LLM. If the task needs a large context, interpretation can live on the host while the device retains threshold rules. Map every requirement to a responsible component and a measurable signal so the architecture diagram contains commitments, not just arrows.

Define host messages as proposals, not commands

Use a versioned structured schema containing device_id, request_id, sequence, deadline, action, typed parameters, and authentication data. On receipt, the ESP32 verifies source, version, deduplication, freshness, the action allowlist, parameter ranges, and current state before converting the message into an internal event. Natural language must never drive GPIO directly. Test duplicate frames, reordering, replay, partial packets, and unknown fields, and make rejection reasons auditable.

Normal and degraded paths must share one state machine

Define explicit states and transitions for online operation, a slow host, link failure, model failure, sensor anomalies, and actuator faults; do not improvise actions inside exception callbacks. After a timeout, cancel the host request, discard late replies, and switch the device to a local threshold or safe-hold mode. Reconnection requires a fresh handshake and sequence synchronization. Inject network loss, restarts, packet loss, host saturation, and watchdog resets to verify that every path returns to a known state.

An edge chip is a heterogeneous memory system, not a TOPS label

The CPU owns control flow, tokenization, sampling, and unsupported operators. A GPU suits regular parallel work but command submission, shader compilation, buffer conversion, and contention with graphics all enter latency. A DSP or NPU can execute supported dtype, shape, and graph patterns at low energy, but may require static dimensions, specific quantization, or vendor compilation. Unified memory removes some explicit PCIe copies; it does not make movement free, because cache coherence, page migration, bandwidth contention, and layout conversion still consume time and energy. Advertised TOPS is usually an arithmetic peak at one low precision and includes neither tokenization, KV management, nor sampling. It also says nothing about operators falling back to CPU. Inspect runtime traces for subgraph placement, copied bytes at every boundary, first-compilation latency, and interference from camera or UI work before explaining TTFT and ITL.

A model file is only part of the delivery contract

GGUF packages tensors, quantization types, and inference metadata for llama.cpp's low-dependency, multi-backend ecosystem. ExecuTorch exports a PyTorch model ahead of time into a .pte artifact and uses partitioners and delegates for target backends. MLC LLM compiles model representations into platform code and parameters. LiteRT-LM assembles tokenizers, decoders, and cross-platform APIs over LiteRT. MNN-LLM connects a mobile runtime, multiple backends, and product integration. A format name does not guarantee equal semantics: tokenizer assets, chat templates, RoPE settings, special tokens, per-tensor scales, KV dtype, and sampling defaults belong to the package contract. Release metadata must capture the source checkpoint, conversion-tool commit, command, hashes, license, supported context, and verification prompts. Validate schema and compatibility before loading so a runtime cannot silently choose a wrong default. Successful parsing proves that bytes are readable, not that output, speed, and permissions satisfy the product.

Treat lowering, partitioning, and fallback as an observable compilation process

On-device deployment usually captures or exports a graph, normalizes operators, fuses patterns, inserts quantize or dequantize boundaries, and asks a partitioner to assign supported subgraphs to CPU, GPU, or NPU delegates. Each boundary may convert layout, dtype, or memory domain. One unsupported operator can fragment a graph into several regions, introduce repeated synchronization and copies, and make an enabled NPU slower than CPU-only execution. Preserve a compiler report: delegated nodes, fallback nodes, subgraph input contracts, workspace, and first-run compilation caches. Dynamic sequence lengths may need shape buckets, chunked prefill, or dedicated decode graphs, with boundary tests. A delegate failure path must be tested rather than merely theoretically runnable; define its capacity limit, timeout, logs, and recoverable model version. Join compiler logs, runtime traces, and system power samples with one request ID.

Accept quality, experience, energy, temperature, and recoverability together

An edge benchmark pins the checkpoint, quantization, prompt suite, backend, threads, power mode, ambient conditions, and software versions. Report cold versus cache-warm loading, prefill throughput and TTFT, decode throughput and ITL, peak RSS, joules per token or average power, device temperature, and frequency after sustained use. Generation speed must remain paired with task quality, format compliance, and refusal policy. Telemetry carries the model hash, delegate-partition summary, and trace ID so a fleet can compare releases. Ship signed model bundles with a compatibility matrix, staged rollout percentage, health gates, and rollback. Drill offline operation, delegate initialization failures, excess temperature, memory exhaustion, and cloud-fallback timeout. The deliverable is not the highest tokens/s from one cool prototype; it is evidence that the target device population maintains experience and safety across real thermal state, networking, and foreground load.

Code or command example

proposal = edge_llm(request)
validated = policy_and_schema_check(proposal)
if validated:
    mcu_execute(validated)
else:
    enter_safe_degraded_state()
record(version_set, latency, energy, thermal, failures)

Hands-on lab

Engineering pitfall

Further reading

• llama.cpp Server (official or primary source)
• Official llama.cpp and GGUF repository (official or primary source)
• Official ExecuTorch LLM deployment guide (official or primary source)

Bridge to the next chapter

The 15-day route is complete. Start the next project from its target device and real workload: write safety and acceptance boundaries first, then select model and runtime, and leave reproducible evidence for every optimization.