Shared APIs versus dedicated deployments

There are two ways to get tokens out of a model. You can send requests to a shared API, where a provider runs a large fleet for many customers at once and bills you per token. Or you can run a dedicated deployment, where GPUs are reserved for your traffic alone and you pay for every hour they exist, busy or idle. The first turns inference into a variable cost that tracks usage. The second turns it into a fixed cost that gets cheaper per token the harder you work the hardware.

Neither is the grown-up choice. Most products start on shared APIs because nothing else is cheaper at low volume, and many keep some workloads there forever. The question is never "which is better" but "at what volume, and for which workload, does the other one win".

What each option gives you

Some workloads decide themselves. Kite's tab completion needs completions faster than a developer types, which is hard to promise from a fleet tuned for everyone, and its next model is a small completion model Kite plans to fine-tune on code, which no shared API hosts. That workload goes dedicated early, even at low volume. Chat, served by the open Qwen/Qwen3-30B-A3B-Instruct-2507, is available from several API providers, so for chat the decision is mostly arithmetic.

Between the two extremes sit managed dedicated endpoints: a provider runs the engine and the autoscaling, and you pay per GPU-hour for replicas that serve only you. They trade some cost for most of the operations work.

Working out the break-even for Kite chat

The cost formula for an API is:

API cost = input tokens × input price + cached input tokens × cached price + output tokens × output price

The prices below are illustrative, not quotes; real prices vary by provider and change often. From the orientation lessons, Kite chat serves 200,000 requests a day, each with 6,000 input tokens and 400 output tokens. Every request starts with the same system prompt and usually the same retrieved files, so assume the 5,700-token prefix hits the provider's prompt cache and only the 300-token question is billed at the full input rate: 95% of input tokens are cached.

Over a 30-day month that is 30 × $133 = $3,990.

The dedicated side uses a different formula: GPU-hours × hourly price. Suppose a load test shows Kite's peak of 12 requests per second needs four replicas of one H100 each, with the model stored in an 8-bit format that a later module covers. At an illustrative $2.50 per GPU-hour on-demand:

4 GPUs × 720 hours × $2.50 = $7,200 per month

At today's traffic the API is cheaper by $3,210 a month, before counting a single hour of engineering time on the dedicated side. That surprises people who compare an API's price per token with a GPU running flat out. The four GPUs are sized for ten in the morning; at three in the morning they are nearly idle, and they still cost $2.50 an hour each.

Now turn it around and ask when the fixed fleet wins. Per request, the API costs $133 ÷ 200,000 = $0.000665. The fleet costs $7,200 a month regardless, so it breaks even at:

$7,200 ÷ $0.000665 ≈ 10.8 million requests per month ≈ 361,000 requests per day

That is 1.8 times today's traffic (361,000 ÷ 200,000). If the same four GPUs could absorb it, every request beyond that point would be cheaper on dedicated hardware. But these four were sized for today's peak, so they cannot: more traffic means more GPUs, and the crossing point moves away with them. The honest way to read the break-even is per GPU. At $0.000665 a request, a $2.50 GPU-hour has to serve $2.50 ÷ $0.000665 ≈ 3,760 requests to match the API, and today's fleet averages 200,000 ÷ (4 × 24) ≈ 2,080. Dedicated hardware wins once each GPU does more work across the day, and techniques such as prefix caching, continuous batching, and speculative decoding exist to raise that number. Measuring it is a benchmarking question, and Module 04 answers it.

Engineering note. Compare the two options over at least a week of real traffic, with nights and weekends included, and add the engineering time a dedicated deployment needs to the GPU bill. A comparison built from one busy afternoon will always make dedicated GPUs look cheaper than they are.

Where it goes wrong

• Pricing GPUs at full utilization. Dividing a GPU's hourly price by its peak throughput gives a cost per token that only exists at peak. Use the average utilization across the whole day.
• Losing the cache discount. The API bill above assumed 95% cached input. A deploy that puts the current timestamp at the top of the system prompt changes the prefix on every request, the cached share collapses, and the bill rises without any change in traffic.
• Rate limits during a launch. A shared API returns HTTP 429 when you exceed your quota. A product launch that triples traffic in an hour can hit that ceiling before anyone approves a limit increase.
• Other customers' traffic. On a shared fleet, your P99 time to first token moves with load you cannot see.
• One dedicated replica. A single replica has no failover. When its GPU fails, and GPUs fail far more often than CPUs, the whole workload is down. Plan for at least two.
• Model retirement. An API provider retires model versions on its own schedule, which can force an unplanned migration and a fresh round of evaluation.

Try it

Recompute Kite's monthly API bill with only 20% of input tokens cached, keeping every other number the same, and decide whether the four-GPU deployment is now cheaper. Then, using your own product's numbers from the orientation exercise, compute cost per request on an API and the requests per day at which a fleet of your chosen size breaks even. (For Kite at 20% cached, 4,800 uncached and 1,200 cached tokens per request: $192 + $12 + $64 = $268 a day, $8,040 a month, so the $7,200 fleet wins.)

Next, latency budgets and workload shapes splits a request's allowed time into parts and shows why Kite's four workloads need different answers.