DigitalOcean Inference Router, Now Cache-Aware: Why the Cheapest Model Isn't Always the Best Deal

DigitalOcean Inference Router, Now Cache-Aware: Why the Cheapest Model Isn't Always the Best Deal | DigitalOcean

© 2026 DigitalOcean, LLC. Sitemap .

Product updates

DigitalOcean Inference Router, Now Cache-Aware: Why the Cheapest Model Isn't Always the Best Deal

By Salman Paracha

Updated: August 20, 2026 10 min read

" , messages=messages, extra_headers= { "X-Model-Affinity": session_id, } , ) assistant_reply = response.choices[0].message.content messages.append( { "role": "assistant", "content": assistant_reply } )

The first request is routed according to the developer’s configured task, model pool, and routing preferences. Requests with the same X-Model-Affinity value are then treated as part of the same unit of work, allowing Inference Router to preserve the session’s model binding and reuse its cached context. Affinity identifiers should represent meaningful units of work: a coding session, research task, support conversation, or individual agent run. When the application starts a genuinely new task, it can provide a new identifier, allowing the Inference Router to make a fresh preference-aware decision.

For common agentic requests, Inference Router can also infer affinity if an explicit identifier is not available. It derives a stable session key from the request context that remains unchanged across turns, including system and developer instructions, tool definitions, and the first user message. If that stable prefix changes, Inference Router treats the cache as cold and establishes a new binding. This reassigns the session to a model, which then starts accumulating its own warm cache from scratch.

Control cache-breaking switches with a routing budget

Model affinity headers are ideal for applications that already track meaningful units of work—such as research tasks and support conversations—and want deterministic control over which requests share the same model binding. With this release, we’ve also introduced the routing budget: a complementary control that keeps Inference Router evaluating alternative models without requiring any changes to your application code.

When the routing policy proposes switching models, Inference Router calculates the incremental cost of leaving the session’s warm cache. It does this by comparing the cached input cost of staying on the current model with the uncached cost of rebuilding the context on the candidate model, then evaluates that cost against the session’s cumulative switching spend.

Developers can define this trade-off with a maximum switching budget, set relative to what the session would have cost had it stayed on the existing model. For example, X-Routing-Max-Switch-Spend-Pct: 20 limits cumulative switching costs to 20% above that baseline. Model selection and economics remain separate: the router identifies its preferred model, while the routing budget determines whether switching to it is worth the additional input cost.

curl -i "https://inference.do-ai.run/v1/chat/completions" \ -H "Authorization: Bearer $MODEL_ACCESS_KEY" \ -H "Content-Type: application/json" \ # X-Model-Affinity is managed by the router if not set explicitly \ -H "X-Routing-Max-Switch-Spend-Pct: 20" \ -d ' { "model": "router:software-engineering", "messages": [ { "role": "user", "content": "Summarize this in 2 sentences." } ] } '

Together, developers can use these controls to choose the appropriate level of involvement:

Use X-Model-Affinity when the application already has an authoritative session or task identifier.

Let Inference Router automatically detect affinity based on stable agent context across related turns.

Configure X-Routing-Max-Switch-Spend-Pct to control how much additional input cost the router can incur by switching models.

New Analyze Page

We have updated the Analyze page to give you detailed visibility into how your router makes cache-aware decisions.

In the top-line router view, you can quickly get answers to questions like:

What is overall caching efficiency?

How many requests switched models?

How many were held to keep the cache warm and what is the overall latency as a result?

This lets you get a high level snapshot of your router at a glance.

From here, you can drill down further into the behavior of specific models and tasks to get a more detailed understanding of the traffic mix. This makes it easier to identify specific model hotspots, validate routing strategy, and tune router preferences over time.

We’ve also added trend tracking for cache efficiency giving you visibility at both request and token level over time. This makes it easier to spot cache regressions, understand performance trends, and validate the impact of prompt and cache tuning changes.

Together, these views help teams move from high-level monitoring to targeted optimization right from the Inference Router UI.

Routing that starts with developer preferences

When we launched DigitalOcean Inference Engine and Inference Router , we gave developers a way to define tasks, create model pools, and express whether they wanted to optimize for quality, cost, or latency. Inference Router then semantically matches each request to a task and applies those preferences to select a model. Developers can start with DigitalOcean presets—opinionated, routinely updated model selections informed by our evaluations—or define their own tasks, model pools, and priorities. Either path works out of the box: no router training or application-side routing logic required. Early customer LawVo reported reducing inference costs by more than 40% while maintaining the accuracy, speed, and reliability its users expected*.

This approach is grounded in years of research into preference-aware routing. In Arch-Router: Aligning LLM Routing with Human Preferences , our team introduced a compact 1.5-billion-parameter model that maps requests to developer-defined domains and actions and can incorporate new models without retraining. We published the model with open weights —the broader approach remains available through Plano , our Apache-licensed open-source AI proxy and data plane. That research originated from a simple observation: benchmarks are useful, but they are not preferences.

Model benchmarks are maps, not routing tables

Benchmarks let us compare models under controlled, repeatable conditions. They help narrow a large model catalog, identify broad strengths, and bootstrap routing before an application has enough real-world traffic to run its own...