Serving Fairness

LLM Serving Fairness North Mini Code. Cohere's first model for developers. Learn more

Jun 17, 2026

7 minute read

LLM Serving Fairness: No more noisy neighbours How Cohere is ensuring every tenant gets their fair share of compute

Running large language models as a multi-tenant SaaS platform comes with a deceptively hard problem: many organizations share the same pool of GPUs, and their traffic is bursty and uneven. Left unmanaged, one customer's spike can become every other customer's latency problem.

In this blog post, we’ll walk you through Cohere’s new solution to scheduling inference requests fairly across tenants, using a combination of architectural patterns and a classic scheduling algorithm. The problem: the noisy neighbor Inference is most efficient when requests are processed in batches. A GPU that’s been fed a full batch runs hot and economically, while one handling a single request at a time mostly sits idle. So, requests queue up briefly and get packed into batches before hitting the hardware.

The catch is ordering . Imagine a naive queue: a single, shared line ordered only by priority and deadline. Now, consider what happens when one organization fires 10,000 requests in a burst while a second organization sends just five. With a single global queue, the 10,000 requests pile in front, and the five well-behaved requests wait at the back.

This is the classic noisy-neighbor problem, and a multi-tenant LLM platform is no different from any other shared system that faces these traffic patterns.

The goal of Serving Fairness is to isolate tenants from one another, so that the share of inference capacity that a tenant receives depends on fair scheduling — not on how aggressively it floods the queue. At the same time, it still preserves priority and deadline ordering within each tenant while maintaining batching efficiency. The solution: A layered approach to capacity management Cohere manages workloads fairly across tenants by combining four distinct mechanisms, each solving a different slice of the problem. They run in a fixed order: a Rate Limiter controls admission on the way in , and then three selectors — Performance Tier , Deficit Round Robin , and Priority — select the next request on the way out.

Here is the architecture and step-by-step flow: 1. Rate Limiter Before a request ever joins the scheduling queue, it passes through admission control . Rate limiting caps the maximum number of inference requests that a single tenant can submit within a given timeframe,such as requests per minute or per month.

At Cohere, these limits are configured at the endpoint level and differ between models based on how many resources each model consumes. For example, a heavyweight generative model carries tighter limits than a lightweight embedding model.

There is also a real-time throttling check: if the queue is already so backlogged that a new request could not be served within its latency target, the request is simply rejected early instead. This protects the system from accepting work it cannot honor and keeps latencies predictable under load.

After requests have been admitted, they proceed to the series of selectors below. 2. Performance Tier (selector one) The first selection decision is the tier . Compute resources are prioritized based on service-level agreements (SLAs): higher-paid tiers are allocated greater processing priority and faster queue handling than lower tiers (or Trial Tiers). In turn, these latter customers are served as capacity allows.

Crucially, tiering determines who goes first ; it does not, by itself, prevent a single large tenant within a tier from dominating. That is the purpose of the next two layers. 3. Deficit Round Robin (selector two) The heart of the system is the Deficit Round Robin (DRR) algorithm, which ensures an equitable distribution of compute across the fleet within a tier .

Each tenant (an "organization") gets its own line. Instead of draining whichever line is longest, the scheduler takes turns between tenants. Each tenant is granted a small budget — a quantum — of work that it may perform on its turn. When a tenant uses its turn, its budget is debited by the cost of the request it just sent through. A tenant that runs out of budget is skipped until its budget is replenished on the following round.

The elegance of DRR is that it is both work-conserving and weighted . Cheap requests let a tenant come up more often, and expensive ones less, so no tenant can run away with the GPU — but no capacity is wasted either. Returning to our earlier example: even if Org A queued 10,000 requests and Org B only five, Org B still gets its due turn every cycle. Org A's burst no longer translates into Org B's latency. What the budget measures The scheme hinges on two key variables: Quantum: How much budget a tenant is granted each round Cost: How much budget each request consumes

The crucial design choice lies in what unit those variables are expressed in. Critically, this determines how we might conceptualize “fairness” in different inference contexts. At Cohere, we use two cost models depending on the endpoint: request-based budgeting and token-based budgeting. Request-based budgeting For simplicity, the cost of every request is given as 1, and the quantum is a count of requests that a tenant may send per turn. Fairness is, therefore, measured purely in the number of requests.

This is suboptimal for generative endpoints , such as chat and completions, where request serving costs can vary dramatically. A request with a 100K-token prompt may consume orders of magnitude more resources than one with a 1K-token prompt. For reasoning-capable models, the total cost depends not only on the input size, but also on the amount of intermediate reasoning, planning, and output generation triggered by the specific context of the request.

Ideally, DRR would use a feedback loop based on the EMA (Exponential Moving Average) of a request's token-normalized serving cost, allowing budgets to adapt to observed resource consumption. Static budgeting works best when requests within an endpoint are broadly similar in size and cost . Token-based budgeting Here, the cost of a request is its token count, and the quantum is a token budget per round. Fairness is now measured in actual work performed. This is the natural fit for batched endpoints like embeddings and rerankers, where the token sum of a batch — not the number of items —...