The life of an inference request

A developer types a question into Kite chat and presses Enter. About four hundred milliseconds later the first word appears, and eight seconds after that the answer is complete. Those two numbers are not mysteries. They are sums: the time spent on the network, the time spent waiting for a GPU, the time spent reading the prompt, and the time spent writing the answer one token at a time.

This lesson follows that one request from end to end and puts a time on every stop. Every later lesson in the course makes one of these stops faster or cheaper, so it is worth knowing the whole route before examining any single part.

Eight stops between the keystroke and the answer

1. The client builds the request. Kite's editor plugin collects the system prompt, a few retrieved files, and the question into a list of messages and sends it to an OpenAI-compatible endpoint with stream=True. If it reuses an open connection, the request leaves immediately; if it opens a new one, the TLS handshake adds tens of milliseconds.
2. The network carries it. Light in fiber covers about 200,000 km per second, so a user 1,500 km from the data center pays at least 7.5 ms each way before any routing overhead.
3. A gateway admits it. It checks the API key, applies rate limits, and chooses a replica. A good router prefers the replica that already holds this conversation's prompt in memory.
4. The engine queues it. The engine (vLLM, SGLang, or TensorRT-LLM) runs many requests together in one batch. If the batch is full or GPU memory is short, the request waits.
5. The prompt becomes tokens. The engine applies the model's chat template, which wraps each message in special marker tokens, and tokenizes the result. Kite's request becomes about 6,000 token IDs.
6. Prefill reads the prompt. One forward pass processes all 6,000 tokens in parallel. This is compute-bound work: the GPU's arithmetic is the limit. Prefill stores intermediate attention results for every token, the KV cache, and produces the first output token.
7. Decode writes the answer. Each further token needs its own forward pass. Every step reads the model's weights and the growing KV cache from GPU memory, samples one token, and appends to the cache. At small batch sizes the limit is how fast memory can be read, not arithmetic.
8. Tokens stream back. Each new token is turned back into text and sent to the client as a server-sent event. Generation stops at an end-of-sequence token or at max_tokens, and the engine frees the request's KV cache for someone else.

Stops 1 to 6 decide when the first token arrives. Stop 7 decides how fast the rest arrive.

Putting times on each stop

Assume these measurements for one Kite chat request on a dedicated deployment of Qwen/Qwen3-30B-A3B-Instruct-2507. They are illustrative, but the arithmetic is the same for any real set.

Time to first token (TTFT) is everything before decode starts:

TTFT = network + queue + prefill = 40 + 60 + 300 = 400 ms

End-to-end latency adds the decode loop:

end-to-end = network + queue + prefill + (output tokens × ITL) = 40 + 60 + 300 + (400 × 20) = 400 + 8,000 = 8,400 ms

The per-user token rate is 1,000 ÷ ITL in milliseconds = 1,000 ÷ 20 = 50 tokens per second.

Look at the proportions. The user sees text after 0.4 seconds, and 95% of the request's life (8,000 of 8,400 ms) is spent in decode. A team that wants Kite to feel faster should attack queue and prefill, which are the parts the user sits and stares at. A team that wants each GPU to serve more users should look at decode, because decode is where a request occupies the GPU longest.

The request also occupies memory for that whole time. The KV cache for one token is 2 × layers × KV heads × head dimension × bytes per value. For Qwen3-30B-A3B (48 layers, 4 KV heads, head dimension 128) in BF16:

2 × 48 × 4 × 128 × 2 = 98,304 bytes = 96 KiB per token

At the end of the answer the request holds 6,000 + 400 = 6,400 tokens, so 6,400 × 96 KiB = 614,400 KiB = 600 MiB. Fifty requests like it running together hold about 29 GiB of KV cache (50 × 600 MiB = 30,000 MiB), on top of the 61 GB of weights. Memory, more than arithmetic, is what limits how many requests fit in a batch.

Engineering note. Give every request an ID at the client and log a timestamp at each hop with it: sent, admitted, scheduled, first token, last token. When someone reports that Kite is slow, you want to answer "queue time tripled at 10:05", not "the model seems slow today".

Where it goes wrong

• The engine says 300 ms, the user says two seconds. Engine metrics start when the engine receives the request. Time in a gateway queue, a cold connection, or a distant region is invisible unless the client measures TTFT too.
• A new connection per request. A client that creates a fresh HTTP client for every call pays a TLS handshake each time, tens of milliseconds added to every TTFT.
• One huge prompt stalls everyone. A 100,000-token prefill occupies the GPU for a long pass, and every streaming request in the same batch pauses. Chunked prefill, on by default in current vLLM, splits long prompts so decode steps can interleave.
• No max_tokens. A model that loops keeps decoding, holding a batch slot and a growing KV cache until it hits the context limit.
• A wrong chat template. Sending raw text where the model expects its template produces rambling, longer answers: more decode time for worse output.

Try it

Repeat the arithmetic for Kite's tab completion, served by meta-llama/Llama-3.1-8B-Instruct: 30 ms of network, 5 ms of queue, a 1,500-token prompt that prefills in 40 ms, and a 20-token completion at 8 ms ITL. Compute TTFT, end-to-end latency, and tokens per second. Then compute the KV cache it holds at the end, using 32 layers, 8 KV heads, and head dimension 128 in BF16. You should get 75 ms, 235 ms, 125 tokens per second, and about 190 MiB (1,520 tokens × 128 KiB).

Next, shared APIs versus dedicated deployments asks whether Kite should run any of this on its own GPUs at all.