What is inference engineering?
Inference engineering is the discipline of running trained AI models in production so that answers are fast, reliable and cheap. It covers model selection and routing, caching, serving on GPUs and the measurement of latency, throughput and cost per token.
What does inference mean?
Inference is the act of using a trained model: tokens go in, the model computes, tokens come out. Training happens once per model version. Inference happens on every request, which makes it the recurring cost of any AI product.
Which metrics define inference performance?
| Metric | Definition |
|---|---|
| Time to first token (TTFT) | Time from sending the request to receiving the first output token |
| Inter-token latency | Average time between consecutive output tokens |
| Tokens per second | Output tokens generated per second, for one request or for the whole system |
| Requests per second | Completed requests per second at a given concurrency |
| End-to-end latency | Total time from the request to the last token |
| Cost per million tokens | Price or infrastructure cost of one million input or output tokens |
The definitions follow NVIDIA’s LLM benchmarking documentation. The metrics trade off against each other: adding concurrency raises total throughput and also latency per request, until the system saturates.
What are the main inference optimization techniques?
| Technique | What it does | Trade-off |
|---|---|---|
| Model routing | Sends each request to the cheapest model that is good enough | Needs evals per task |
| Prompt caching | Reuses the computed prefix of repeated prompts | Only helps when the prefix is stable |
| Continuous batching | Merges requests on the GPU as they arrive | Batch limits must be tuned against latency |
| Quantization | Stores weights at lower precision | Possible quality loss, to be measured |
| Speculative decoding | A draft model proposes tokens and the target model verifies them | Needs a well-matched draft model |
| KV-cache management | Reuses attention state between tokens and requests | Memory pressure on the GPU |
Speculative decoding was introduced by Leviathan et al., who showed it speeds up generation without changing the model’s outputs. PagedAttention, the memory manager behind vLLM, allocates the key-value cache in pages; its authors report 2–4× higher throughput at the same latency than earlier serving systems.
Where does inference cost come from?
- Input tokens: system prompts, retrieved context and conversation history, sent again on every call.
- Output tokens: usually priced higher than input tokens.
- Agent loops: one task can mean dozens of model calls.
- Idle capacity: self-hosted GPUs cost the same whether they are busy or not.
- Retries and failures: work that is paid for twice.
What does an inference engineer do?
- Instrument traffic: tokens, latency and cost per feature and per request.
- Build the cost model: where the spend goes and what each lever is worth.
- Choose models and routing, backed by evals for each task.
- Design the serving stack: gateway, caching, model servers and autoscaling.
- Load-test against real traffic patterns before launch.
- Keep tuning as models, prices and traffic change.
Frequently asked questions
Is inference engineering the same as MLOps?
They overlap, but the focus differs. MLOps covers the lifecycle of training and deploying models. Inference engineering focuses on serving: the latency, throughput and cost of models in use, including models consumed through provider APIs.
What is a good time to first token?
It depends on the product. Interactive chat and voice need the first token quickly because a person is waiting. Background agents can trade time to first token for throughput and cost. Set the target from the user experience, then engineer to it.
Does inference engineering only apply to self-hosted models?
No. Routing, prompt caching, context size and output length decide cost and latency on provider APIs too.
Sources
- NVIDIA — LLM inference benchmarking: metrics
- NVIDIA — LLM inference benchmarking: fundamental concepts
- Kwon et al. — Efficient Memory Management for LLM Serving with PagedAttention (vLLM, 2023)
- Leviathan et al. — Fast Inference from Transformers via Speculative Decoding (2022)
- OpenAI — Prompt caching
- Anthropic — Prompt caching