Resources

Release notes and deep dives from the engine most teams start with.

SGLang internals, large MoE serving and benchmark write-ups.

TensorRT-LLM, Dynamo, FP8 and FP4 results, straight from the hardware vendor.

Model releases, quantization guides and serving recipes.

Every GPU term, from CUDA kernels to NVLink, explained in plain words.

Production inference notes on speculation, disaggregation and cold starts.

Chip Huyen

Sebastian Raschka

Wen-mei Hwu, David Kirk, Izzat El Hajj

Martin Kleppmann

Betsy Beyer et al.

Inference engine · PagedAttention, continuous batching and an OpenAI-compatible server.

Inference engine · RadixAttention prefix caching and strong large-MoE serving.

Inference engine · NVIDIA's engine with hand-tuned kernels and FP8/FP4 support.

Orchestration · KV-aware routing and disaggregated prefill and decode across engines.

Kernels · Exact attention computed in tiles that fit on-chip memory.

Kernels · Attention and sampling kernels built for LLM serving.

Quantization · Post-training FP8 and NVFP4 quantization that every engine can load.

KV cache · Offload and share KV cache across CPU, disk and replicas.

Local inference · GGUF models on laptops and desktops, CPU and GPU.

YouTube · Community lectures on CUDA, Triton kernels and inference performance.

YouTube · Conference talks on torch.compile, vLLM and serving at scale.

YouTube · GTC sessions on GPUs, TensorRT-LLM and Dynamo.

YouTube · Build a GPT from nothing and see exactly what runs at inference.

Podcast · Interviews with the people building inference platforms.