Serve AI models fast and cheap,
from GPU to production.

Track your progress.

Sign inor start with What inference engineering is

Full curriculum

OrientationIE2

Orientation

What inference engineering is

The life of an inference request

Module014

Before You Optimize

Shared APIs versus dedicated deployments

Latency budgets and workload shapes

Measuring TTFT, TPS, and percentiles

Picking the smallest model that passes

Module027

How Models Run

Matmul and activations for inference engineers

Tokens, chat templates, and sampling

Prefill and decode

Transformer blocks and attention

The KV cache and its memory bill

Mixture of experts models

How image and video models generate

Module035

Bottlenecks and Hardware

Compute-bound or memory-bound

Inside a datacenter GPU

Choosing a GPU for a workload

Nodes, interconnects, and MIG

Making attention cheaper

Module045

The Software Stack

CUDA kernels, selection, and fusion

PyTorch, compile, and model files

vLLM, SGLang, and TensorRT-LLM

Continuous batching

Benchmarking and profiling inference

Module054

Quantization and Speculation

Number formats for inference

Quantizing without losing quality

Speculative decoding

EAGLE, Medusa, and n-gram drafts

Module064

Caching, Parallelism, and Disaggregation

Prefix caching

KV cache tiers and cache-aware routing

Tensor, expert, and pipeline parallelism

Disaggregated prefill and decode

Module073

Beyond Text

Vision language and embedding models

Speech pipelines: ASR and TTS

Faster image and video generation

Module086

Inference in Production

Containers and cold starts

Autoscaling, routing, and queues

Multi-cloud capacity and reliability

Shipping changes, cost, and observability

Client code, streaming, and protocols

Capstone: an inference stack for Kite

The free lessons are complete lessons, not teasers. Pro unlocks the hardware, software stack, performance techniques, other modalities and production modules.Read a free lesson