Benchmarking and profiling inference

Every optimization in the rest of this course is a claim that something got faster or cheaper. Two tools turn the claim into a number. A benchmark treats the server as a black box: it sends a realistic stream of requests and records time to first token, inter-token latency and throughput at the percentiles users feel. A profiler opens the box: it records every kernel the GPU ran during a few steps and tells you which ones took the time.

Use them in that order. The benchmark tells you whether there is a problem and how big it is. The profiler tells you why. Profiling a server that already meets its targets at the load you need is a pleasant afternoon with no payoff.