LLM API vs. self-hosted inference: how to decide
Use a provider API by default, and move a workload to self-hosted open-weight models when volume is steady enough to keep GPUs busy, when data must stay in your environment, or when you need latency or customization an API cannot offer.
What is the difference between API and self-hosted inference?
| Provider API | Self-hosted | |
|---|---|---|
| Pricing | Per token | Per GPU-hour |
| Models | Frontier proprietary models and hosted open models | Open-weight models you deploy |
| Capacity | The provider’s problem, within rate limits | Yours to plan and scale |
| Data | Leaves your environment under the provider’s terms | Stays in your environment |
| Operations | None | Serving stack, upgrades, on call |
How do the costs compare?
An API charges for tokens used. A GPU charges for time, used or not. The cost per token of a self-hosted model is the GPU price per hour divided by the tokens it actually serves in that hour, so utilization decides everything.
cost per 1M tokens = GPU cost per hour ÷ (tokens per second × 3,600 × utilization) × 1,000,000Run the formula with your measured throughput and a realistic utilization, not a peak benchmark, and compare the result with the API price of a model of similar quality.
When is a provider API the better choice?
- Traffic is spiky or still small.
- The task needs frontier model quality.
- The team has no capacity to operate GPUs.
- You are still finding out which model the task needs.
When does self-hosting make sense?
- Steady, high token volume on a task that a smaller open-weight model handles well.
- Data residency or contractual rules that keep data in your environment.
- Latency targets that need dedicated capacity close to users.
- A fine-tuned or specialised model.
Can you combine both?
Most production systems do. A gateway routes each request: a frontier model through an API for hard steps, a self-hosted model for high-volume, well-understood steps, and the API as fallback when self-hosted capacity is full.
What should you measure before deciding?
- Tokens per day per feature, input and output separately.
- Concurrency at peak and off-peak.
- Latency targets: time to first token and end to end.
- Quality: eval results of candidate open-weight models on your task.
- Operating cost: who is on call and what an hour of downtime costs.
Frequently asked questions
Is self-hosting always cheaper at scale?
No. It is cheaper only when GPUs stay busy and an open-weight model meets the quality bar. Low utilization, or a quality gap that forces a larger model, can erase the saving.
What is GPU utilization?
Utilization is the share of time a GPU spends serving requests. Because a GPU costs the same idle or busy, low utilization raises the cost of every token it produces.
Which inference server should we use?
Common choices are vLLM, Hugging Face TGI and NVIDIA Triton with TensorRT-LLM. The right one depends on the model, the hardware and features such as structured output or multi-model serving.