01One request, two very different phases

Generating an answer from a language model happens in two phases, and they behave so differently that almost every optimization targets one or the other. Prefill reads your entire prompt at once: every input token is processed in parallel in a single pass, which keeps the GPU's math units busy — it is compute-bound. Decode then generates the reply one token at a time, feeding each new token back in to produce the next. Each decode step does very little math but has to re-read the model's weights from memory, so it is starved for memory bandwidth rather than compute. That single split — a fast, compute-heavy prefill followed by a slow, bandwidth-limited decode — is the key to understanding everything else here.

• Prefill is parallel and compute-bound; it sets how long you wait for the first token (TTFT).
• Decode is sequential and memory-bandwidth-bound; it sets the speed of every following token (TPOT).
• Most serving optimizations make sense only once you know which phase they help.

02The KV cache: don't redo work you've already done

During decode, the model's attention step needs the key and value vectors for every token seen so far. Re-computing those from scratch at every step would be enormously wasteful, so they are stored in a key-value (KV) cache and reused. The catch: the cache grows linearly with the sequence length and the number of requests in the batch, and it is usually the largest non-weight consumer of GPU memory during serving. Whoever controls KV-cache memory controls how many requests you can serve at once.

Two families of techniques help. The first changes how the cache is laid out: PagedAttention (the core of the vLLM engine) splits the cache into fixed-size blocks that can live non-contiguously in memory — the same trick operating systems use for virtual-memory paging — which cuts fragmentation and lets requests share blocks. The second shrinks the cache itself: multi-query and grouped-query attention let many query heads share a smaller set of key/value heads, while eviction policies (H2O) and low-precision quantization (KIVI, low-bit GGUF) trade a little quality for a much smaller footprint — and a smaller cache means room for larger batches.

• The KV cache stores past key/value vectors so the model never recomputes them — but it grows with sequence length and batch size.
• PagedAttention manages the cache like OS memory pages: less fragmentation, flexible sharing across requests.
• MQA/GQA shrink the cache by sharing K/V heads; eviction and quantization shrink it further, at some quality cost.

03Sandbox: toggle the optimizations, watch the numbers move

Below is a single request being served, drawn as a GPU timeline: a short compute-bound prefill block, then a long string of memory-bound decode steps. Flip each optimization on and off and watch the latency and throughput estimates respond. The numbers are illustrative — they show the direction and rough shape of each technique's effect, not a benchmark of any specific model or GPU.

• KV cache mainly speeds up per-token decode (TPOT) by avoiding recomputation.
• Continuous batching packs many requests onto the GPU, lifting throughput far more than single-request latency.
• Speculative decoding verifies several drafted tokens per target pass, lowering TPOT without changing the output.

04Continuous batching: keep the GPU full

Because decode is memory-bandwidth-bound, running a single request leaves most of the GPU idle. The fix is to serve many requests at once. Naive (static) batching waits for a whole group of requests to finish before starting the next group — so the entire batch is held hostage by its longest sequence, wasting GPU time as shorter ones sit done-but-waiting. Continuous batching (also called iteration-level scheduling or in-flight batching, introduced by the Orca system) instead schedules work one decode step at a time: as soon as any sequence finishes, a new request takes its slot. The batch is continuously refilled, GPU idle time drops, and throughput rises sharply — which is why production engines such as vLLM, TensorRT-LLM, and TGI build on this idea.

There is one more wrinkle. Prefill is compute-bound and decode is memory-bound, so mixing them well matters. Chunked prefill (SARATHI) splits a long prompt into chunks and piggybacks ongoing decode steps onto them to keep both resources busy, while disaggregation (DistServe) goes further and runs prefill and decode on different GPUs so they stop interfering with each other and each phase can be tuned independently.

• Static batching wastes time: the batch waits for its longest sequence to finish.
• Continuous batching refills slots per decode step, cutting idle time and lifting throughput.
• Chunked prefill and prefill/decode disaggregation stop the two phases from starving each other.

05Speculative decoding — and exact vs. approximate

Decode is slow because each step produces just one token yet still pays to read the whole model. Speculative decoding exploits this: a small, fast draft model proposes several candidate tokens, and the large target model verifies them all at once in a single forward pass — which in the bandwidth-bound decode regime costs roughly the same as generating one token. A modified rejection-sampling step guarantees the result matches what the target model would have produced on its own, so the output distribution is preserved exactly. A related approach, Medusa, drops the separate draft model and instead adds extra decoding heads that predict several future tokens in parallel, verified together with tree-based attention.

One distinction is worth holding onto. Some optimizations are exact — speculative decoding, PagedAttention, and FlashAttention (an IO-aware attention kernel that tiles work to minimize slow memory reads) change speed and memory only, never the output. Others are approximate — KV-cache eviction (H2O) and low-bit quantization (KIVI) shrink memory by discarding or compressing information, which can shift quality. Knowing which bucket a technique falls into tells you whether to expect identical answers or a quality trade-off.

• Speculative decoding drafts several tokens cheaply and verifies them in parallel; the output is exactly the target model's.
• Medusa reaches the same goal with extra decoding heads instead of a separate draft model.
• Exact methods (speculative decoding, PagedAttention, FlashAttention) never change outputs; approximate methods (eviction, quantization) can.

06Knowledge check