Capstone: an inference stack for Kite

Kite has four model workloads, and each one has pulled a different lesson out of this course. Chat needs a short time to first token over long, repeated context. Tab completion needs an answer faster than a person types. Nightly indexing needs cheap tokens and does not care when they arrive. Voice needs a pipeline of three models that never makes a speaker wait. A design for Kite is not one clever technique. It is a row per workload that turns a latency target and a traffic number into a model, an engine configuration, a replica count and a failure plan, and then adds the rows into one bill.

This lesson writes those rows out. Every traffic and throughput number below is an assumption to be replaced by a benchmark on your own traffic; the method is the part to keep.