Prefill and decode

An LLM request is not one job. It is two jobs with opposite characters glued together. First comes prefill: the model reads the entire prompt in a single forward pass, all tokens side by side, computes the attention keys and values for each one, and produces the first output token. Then comes decode: the model generates one token, appends it, runs another full forward pass for the next token, and repeats until it emits a stop token or hits the limit.

Prefill is limited by how fast the GPU can do arithmetic. Decode is limited by how fast it can read memory. Almost every serving decision in this course, from batch size to GPU choice to splitting the two phases onto different machines, comes back to that difference.

Prefill: lots of math, one read

Take Kite's tab completion model, Llama 3.1 8B in BF16. From the previous lessons: about 8 billion parameters, 16 GB of weights, and about 2 × 8 billion = 16 billion floating-point operations (FLOPs) per token.

A completion request carries a 1,500-token prompt: the current file and a few neighbors. Prefill processes all 1,500 positions in one pass, so the work is 2 × 8 billion × 1,500 = 24 trillion FLOPs.

The weights are still read once for that pass. The GPU's matrix-multiply kernels get large matrices with 1,500 rows and plenty to chew on, so its arithmetic cores are busy, not waiting. Doubling the prompt roughly doubles prefill time (a bit more at long lengths, because attention grows faster than linearly). That is what compute-bound means: the bottleneck is operations per second.

Prefill also leaves something behind. The keys and values it computed for every prompt token are stored in the KV cache, so decode never has to process the prompt again. A later lesson prices that cache.

Decode: one token, every weight

Now the model writes. Each decode step feeds in one new token and produces one more. The arithmetic is modest: 16 billion FLOPs, one fifteen-hundredth of the prefill. But generating that single token still multiplies through every layer, which means reading all 16 GB of weights from GPU memory into the compute cores.

That gives an upper bound on single-request speed: decode tokens per second is at most the memory bandwidth divided by the bytes of weights read per token.

Suppose, for round numbers, a GPU moves 2,000 GB per second between its memory and its cores (an illustrative figure, not a specific product). Then:

• BF16 weights: 2,000 ÷ 16 = 125 tokens/s, so about 8 ms per token.
• FP8 weights: 2,000 ÷ 8 = 250 tokens/s, about 4 ms per token.

The bound ignores KV cache reads and overhead, so real numbers come in lower. Notice what does not appear in it: FLOPs. A GPU with twice the compute but the same memory bandwidth barely speeds up single-user decode. At batch size 1 the cores spend most of each step waiting for weights to arrive.

Batching is the fix. If 32 requests decode together, one read of the weights produces 32 tokens. The per-user speed drops a little, total throughput rises a lot, and decode starts to use the compute it was wasting. Continuous batching, covered in its own lesson, is how engines do this with requests that start and finish at different times.

Which phase owns which number

Put the phases on a clock. End-to-end latency is network time, plus time in the queue, plus prefill, plus output tokens times inter-token latency (ITL). Using illustrative numbers for a Kite completion on the GPU above, with prefill running at 10,000 tokens per second:

That misses Kite's 300 ms P90 budget for tab completion by 60 ms, and the table says where to look. Prefill is the largest line, because this illustrative GPU prefills slowly; a card with more compute, or prefix caching of the file the user has not changed, shrinks it. Decode already runs at the bandwidth bound, so it has nothing left to give at batch size 1.

Change the shape and the answer changes. Give the same model a chat-shaped request like Kite's, which sends 6,000 tokens of system prompt and retrieved files and streams back 400 tokens: prefill becomes 600 ms and dominates what the user feels first, while 400 × 8 ms = 3.2 seconds of decode is spread across a stream they are already reading. Nightly indexing with an embedding model is prefill only: there is no decode at all. An agent that writes a long file from a short instruction is almost pure decode.

Per-user speed follows from ITL: 1,000 ÷ 8 ms = 125 tokens per second for that user. System throughput is a different number, the sum over every request in the batch, and the two pull against each other as the batch grows.

Engineering note. When someone reports "the model is slow", ask which number: time to first token or time between tokens. They have different causes and different fixes, and a single end-to-end average hides which one moved.

Where it goes wrong

• One long prompt stalls everyone. A 30,000-token prompt arrives while 40 users are streaming. If the engine runs that prefill as one step, every stream freezes for its duration. Chunked prefill, on by default in current vLLM, splits it so decode steps interleave.
• Blaming prefill for queueing. TTFT climbs at peak hours while prefill time per token is flat. The requests are waiting, not computing; the fix is capacity or admission, not a faster kernel.
• A throughput number with no shape. "5,000 tokens per second" means nothing without input and output lengths. Prefill tokens are cheap per token and decode tokens are expensive, so a prompt-heavy benchmark flatters any system.
• Buying compute for a decode problem. A single-user, long-output workload on a GPU with more FLOPs but similar bandwidth gets little faster. Check the bottleneck before the invoice.

Try it

On a rented GPU running vllm serve meta-llama/Llama-3.1-8B-Instruct, benchmark two opposite shapes and compare their percentiles:

vllm bench serve --model meta-llama/Llama-3.1-8B-Instruct --dataset-name random --random-input-len 2000 --random-output-len 200 --num-prompts 500 --request-rate 8 --percentile-metrics ttft,tpot,itl,e2el --metric-percentiles 50,90,99

Run it again with --random-input-len 200 --random-output-len 2000. Predict before you look: which run has the higher time to first token, which has the higher end-to-end latency, and does time per output token change much between them?

Next is Transformer blocks and attention, which opens up the forward pass that both phases run.