Agentic Inference in Production: The Four Infrastructure Decisions That Matter

Agentic Inference in Production: 4 Key Decisions | CoreWeave

Announcement

Webinar

Podcast

GTC 2026

CoreWeave recognized as a Visionary in the Gartner® Magic Quadrant™ for Cloud AI Infrastructure. Read the report

Products

Data and storage

Infrastructure control

Runtime acceleration

Model and agent development

Mission control

Solutions

Pricing

Resources

About us

Contact us Login

Contact us Login

Clear

This is the third and final chapter of our agentic inference series, and it’s the one that gets practical. In the first blog, we explored the unique shape of an agentic inference workflow , in which an agent turns one request into a chain of inference calls. The second blog examined what those repeated requests actually contain and how infrastructure can reduce time-to-first-token (TTFT) through prefix-aware routing . Both left the same question open: What does infrastructure built for all of this actually look like once an agent is live and serving real traffic? And which deployment model do you use for it? The answer is different for each team and each agent, depending on what you choose to optimize for, route on, scale on, and measure. Those decisions lead to which deployment model best suits your agent. What follows are four decisions that determine whether an agent holds up in production, six questions about your own traffic that answer them, and the three deployment models those answers point to. Four decisions that determine whether your agent holds up Four decisions sit underneath every production agent’s inference stack, whether or not anyone made them deliberately: What you optimize for , what you route on, what you scale on , and what you measure . 1. What you optimize for Your stack targets one of two clocks, and the choice sets your batching and scheduling policy. Time-to-first-token (TTFT): the wait before output starts streaming. The default in most stacks, and the right service-level objective (SLO) when a person is reading along. End-to-end completion: time-to-last-token per step, and time-to-task-completion across the chain. The right SLO when a tool executor is the consumer, because it can’t act on a partial output.

Agentic inference in production comes down to four decisions: what you optimize for, route on, scale on, and count. Here's how each shapes your deployment model. 2. What you route on Three signals are available to your router, and the one you pick decides how often a request lands on a worker that can skip prefill. Least-connections (aka round-robin): sends each request to the first available worker, blind to both the session and the cache. The default in most stacks, and a good choice if your prefix changes on every call. Session identity: pins a conversation to one worker. Cheap to implement, but load-blind, and it captures none of the reuse across different sessions that share a system prompt and tool definitions. Prefix overlap: sends each request to a worker already holding reusable KV state for its prefix, balanced against that worker’s load, so prefill covers only the new tokens. Pays in proportion to how long and how stable your prefixes are.

Blog two calls the third one prefix-aware routing; serving-engine and product documentation often call the same mechanism KV cache-aware routing. 3. What you scale on Two settings here, and they interact. The trigger is what your inference autoscaler watches: Latency or throughput: symptoms. Scaling on them is always late, because by the time p95 moves, the queue that caused it has been building for a while. Concurrent running and queued requests: the demand pressure underneath those symptoms, and the earlier signal.

The floor is how little you keep warm between bursts: Scale to zero: right for a workload that runs a few times a day, punishing for agentic inference traffic. A cold replica has a cold cache, so every resume pays twice: once for the cold start, once for a full prefill of context that was warm twenty minutes ago. A warm replica floor: you pay for idle capacity through the troughs and skip both penalties on every resume. Fixed capacity: nothing to tune, and you pay for your peak all day.

How finely you can set any of this depends on your deployment model. Routing decides where your traffic goes; the floor decides whether there’s anywhere good to send it.

The idle capacity under the warm replica floor is what you pay to avoid a cold start, plus a full prefill on every burst resume. 4. What you measure Two units, and they can move in opposite directions. Cost per token: what you’re billed on, and what most dashboards show by default. In a market where per-token prices keep falling, it’s a flattering number that can improve while your bill grows. Cost per task: the whole chain. Tokens per step × steps per task × reasoning overhead, with every cache miss repaying for context the system already processed once.

These four decisions don’t have one universal answer. You answer them differently depending on how much of the serving stack you own, which is what the rest of this piece is about. What shape is your inference traffic? Workload shape comes before configuration. Prefix-aware routing, cache affinity, speculative decoding, disaggregated prefill—none of these are universally required, and each pays off in a particular regime. Speculative decoding fades as batch sizes climb. Disaggregation degrades below a scale threshold. Caching, as we'll get to, has a ceiling that a single mutating token can put you under. Techniques pitched without their boundaries are the fastest way to spend engineering time on something your traffic was never going to reward. Know your traffic shape first; then decide which parts of the stack are worth configuring. Six questions get you most of the way there. 1. How stable is your prefix? Caching requires an identical prefix, not a similar one. A timestamp in the system prompt, a reordered tool definition, a session variable injected before the instructions: any of them is a full miss on every call. Stable prefixes earn high hit rates; mutating ones collapse to nearly none. That exact-match ceiling makes prompt construction an infrastructure decision as much as a prompt-engineering one. 2. How long do your chains run? The more steps per task, the more the whole-chain SLO from Section 1 governs, and the more each step's latency compounds into something the user actually feels. Two hundred milliseconds per step is invisible at three...