LLM Performance and Infrastructure

Learning goal

Chapter keywords

Bridge from the previous chapter

Day 13 proved which backends a request actually traversed. This chapter upgrades that correctness baseline into a cross-scale performance ledger, extending device measurements into cluster communication, state scheduling, and service objectives.

Chapter walkthrough

Write the experiment contract before starting the timer

A comparable benchmark fixes at least the model and hash, quantization format, runtime commit, threads and CPU affinity, prompt-token count, output limit, sampling, concurrency, power supply, and cooling. Warm up first, run enough repetitions, and retain raw samples. If two tests use different input lengths or temperatures, their p95 and tokens/s are not directly comparable. Save these fields as a machine-readable manifest so firmware or model upgrades can rerun the test automatically.

Diagnose TTFT and generation speed separately

TTFT includes queuing, templating, tokenization, context setup, prefill, and the first sampling round. Steady-state decode mainly reflects incremental reads of weights and KV, small matrix operations, and sampling. A long prompt raises the former; a long context and bandwidth pressure slow the latter. Place monotonic-clock timestamps on the same request and report prompt tokens/s, the inter-token-latency distribution, and output tokens/s rather than one total duration.

Let a bottleneck hypothesis drive the next observation

Low CPU utilization does not prove that compute is abundant: threads may be waiting on memory, a lock, GPU synchronization, or network backpressure. Use a trace to find the longest stage and then state a falsifiable hypothesis. If bandwidth is suspected, change quantization or context and observe speed and bytes moved; for compute, vary core count or frequency; for copies, record partition boundaries. Change one variable at a time so the evidence can distinguish correlation from causation.

Draw the data-movement hierarchy before discussing compute

A kernel's hottest scalar may remain in a register, while a thread block reuses tiles in on-chip SRAM or shared memory. Weights, activations, and KV cache primarily occupy HBM. Cross-accelerator tensors traverse PCIe or a dedicated scale-up fabric; cross-node collectives traverse NICs and switches; checkpoints finally reach local NVMe or remote object and parallel file systems. Moving outward generally offers more capacity but raises latency, energy, and contention. Arithmetic intensity asks how much computation is performed for every byte moved, and attention, MoE dispatch, and embedding lookup can have radically different limits. A useful engineering diagram labels every edge with bytes, frequency, contenders, and topology instead of saying only that the GPU is fast. HBM capacity determines whether state fits, HBM bandwidth caps many token-by-token kernels, the intra-node fabric shapes TP collectives, and the scale-out network shapes DP, EP, and recovery. Peak FLOPS becomes useful throughput only when operands arrive in time, kernel shapes use the machine well, and communication can be hidden.

Six parallel dimensions cut different axes of the same training graph

Data parallelism replicates the model, partitions the input batch, and reduces gradients after backward. Tensor parallelism cuts matrices or attention heads inside a layer, so each layer may need an all-reduce or reduce-scatter/all-gather pair; that fine-grained critical path prefers a low-latency, high-bandwidth scale-up domain. Pipeline parallelism assigns contiguous layers to stages, moves boundary activations, and feeds microbatches through the pipeline; bubbles and stage imbalance are its central costs. Context parallelism partitions a long sequence and exchanges the keys, values, or intermediates required for attention. Expert parallelism distributes MoE experts and moves routed tokens with all-to-all dispatch and combine; routing skew lets a few experts stall everyone. FSDP/ZeRO partitions parameters, gradients, and optimizer states: gather parameters for computation, reduce-scatter gradients afterward, then release full replicas. These are not mutually exclusive toggles. Production jobs combine dimensions in a device mesh, but every dimension adds layout transitions, failure surface, and tuning choices.

Online inference has two phases, two latency families, and growing state

Prefill processes prompt tokens in parallel and often forms large matrix multiplies; it strongly influences Time To First Token. Decode emits a small number of tokens per iteration but repeatedly reads layer weights and historical keys and values; memory bandwidth and scheduling overhead often dominate, while Inter-Token Latency determines streaming feel. KV cache avoids recomputing attention for old tokens, but capacity grows with concurrency, context length, layers, KV heads, head dimension, and dtype. PagedAttention maps a logically contiguous sequence to physical blocks, reducing external fragmentation and enabling controlled sharing. Prefix caches reuse identical leading blocks, but hits depend on normalized templates, tenant isolation, and routing locality. Continuous batching admits and retires requests at decode iterations, eliminating empty slots left by static batches while turning queuing, preemption, and fairness into first-class decisions. Throughput must be reported under a TTFT/ITL SLO; tokens produced after a request's deadline are not useful serving capacity.

Close capacity planning with SLOs, observability, and failure drills

At ingress, record input and output token counts, model and tokenizer versions, sampling parameters, tenant, deadline, and trace ID. The router records its selection reason, prefix hit, and queue estimate. Workers expose TTFT, per-token ITL, batch width, KV block use, preemption, OOM, and errors. Network telemetry covers collective or point-to-point bytes, congestion, and retries. Capacity models must separate short chat, long-context, batch, and multi-turn traffic because average arrival rate hides the impact of long prompts on prefill and KV. Load tests should replay arrival processes and length distributions, then report p50, p95, p99, and SLO goodput. Drills should kill workers, throttle networks, fill KV capacity, corrupt a checkpoint shard, and roll through an incompatible runtime change. Peak benchmark numbers become operational evidence only when the service rejects, degrades, rolls back, and leaves an auditable trace under these conditions.

Code or command example

record(model, quant, prompt_tokens, output_tokens, hardware, temperature)
measure(TTFT, ITL_p50, ITL_p95, tokens_per_s, joules_per_token)
# At cluster scale, add collectives, KV occupancy, goodput, and recovery

Hands-on lab

Engineering pitfall

Further reading

• NVIDIA GenAI-Perf Metrics (official or primary source)
• Megatron-LM: parallel training of large Transformers (official or primary source)
• The vLLM and PagedAttention paper (official or primary source)

Bridge to the next chapter

The final chapter closes model packages, runtimes, heterogeneous backends, MCU/host safety boundaries, energy, fallback, OTA, and observability into a deliverable edge-LLM product.

Next: Day 15 · Shipping Edge LLMs