Inference engineering

Inference, engineered for cost and speed.

Inference engineering is the work of running models in production fast, reliably and at the lowest cost per useful answer. RXY Studio designs and operates the full path between a request and the GPU: gateways that route each request to the right model, caching, and serving stacks for open-weight models, measured on time to first token, throughput and cost per million tokens.

By Urban Herak, founder of RXY Studio · Updated November 10, 2026

What is inference engineering?

Training makes a model. Inference is every time it is used. For a company running agents, inference is the recurring cost and the latency users feel: each agent step is one or more model calls, and a single task can take dozens.

Inference engineering treats that path as a system to be designed: which model answers which request, what gets cached, how requests are batched on a GPU, and how the whole thing scales when traffic changes.

Which inference metrics matter?

Metric	What it measures	Why it matters
Time to first token (TTFT)	Delay between sending a request and receiving the first output token	What a user perceives as responsiveness
Tokens per second	Output speed once generation starts, per request and across the system	How long answers and agent steps take, and total capacity
Cost per million tokens	What input and output tokens cost on a given model and stack	The unit economics of every feature
p95 and p99 latency	Latency of the slowest 5% and 1% of requests	Tail latency compounds across multi-step agents
Task success rate	Share of tasks completed correctly on an eval suite	Cheap tokens are worthless if the answer is wrong

How do you reduce inference cost and latency?

Routing: send each request to the smallest model that passes the evals for that task, with fallbacks.
Prompt caching: reuse the processed prefix of long, repeated prompts. Major providers bill cached input tokens at a discount.
Response caching: answer repeated questions without calling a model at all.
Continuous batching: the server merges concurrent requests so GPUs stay busy.
Quantization: lower-precision weights cut memory and raise throughput, at a quality cost that has to be measured.
Speculative decoding: a small draft model proposes tokens that the large model verifies in parallel.
Context discipline: shorter prompts and outputs are the cheapest optimization there is.

Provider API or your own GPUs?

Provider APIs are the right default: no capacity planning, frontier models, pay per token. Running open-weight models on your own or rented GPUs starts to make sense with steady high volume, strict data residency, latency targets an API cannot meet, or a fine-tuned model.

The decision is arithmetic. We model both paths on your real traffic: tokens per day, concurrency, latency targets and utilization. A GPU that sits idle is the most expensive token you can buy.

	Provider API	Self-hosted open weights
Cost model	Per token	Per GPU-hour; cost per token depends on utilization
Best for	Variable traffic, frontier quality	Steady volume, data control, custom models
You operate	Nothing	Serving stack, autoscaling, upgrades

What does a serving stack look like?

For open-weight models we build on inference servers such as vLLM, Hugging Face TGI or NVIDIA Triton with TensorRT-LLM, on NVIDIA GPUs from providers such as RunPod, Modal or the major clouds. vLLM is built around PagedAttention, a paging-style memory manager for the attention key-value cache; its authors report 2–4× higher throughput at the same latency compared with earlier serving systems.

Gateway: authentication, routing, rate limits, budgets and fallbacks.
Model servers with continuous batching and streaming.
Autoscaling on queue depth and GPU utilization, with scale-to-zero where cold starts allow.
Observability: tokens, latency and cost per request, traced end to end.
Load tests that reproduce your traffic before launch.

How does an engagement work?

Measure: instrument current traffic for tokens, latency and cost per feature.
Model: find where the spend goes and what each lever is worth.
Build: gateway, caching and, where it pays, a self-hosted serving stack.
Prove: load tests and evals before any traffic moves.
Operate: budgets, alerts and re-tuning as models and prices change.

What you get

Inference audit

Where tokens, latency and money go today, per feature.

Model gateway

Routing, fallbacks, budgets and caching in front of every model.

Serving stack

Open-weight models on GPUs, tuned and autoscaled.

Cost model

API versus self-hosted, calculated on your real traffic.

Dashboards

TTFT, throughput, tail latency and cost per request.

Frequently asked questions

What is time to first token (TTFT)?

Time to first token is the delay between sending a request and receiving the first output token. It is the number users feel as responsiveness, and it depends on queueing, prompt length and the model.

How do you reduce LLM costs without losing quality?

By routing each request to the smallest model that passes the evals, caching repeated prompts and answers, trimming context and batching. Every change is checked against an eval suite, so quality is measured instead of assumed.

When should a company self-host models?

When volume is steady and high, when data must stay in your environment, when you need latency an API cannot give, or when you run a fine-tuned model. We model the cost of both paths on your traffic first.

Which inference servers and GPUs do you use?

We choose per workload: servers such as vLLM, TGI or Triton with TensorRT-LLM, on NVIDIA GPUs from providers such as RunPod, Modal or the major clouds.

Do you work with provider APIs too?

Yes. Most systems mix both: frontier models through provider APIs and smaller open-weight models for high-volume steps.