How To Optimize Llm Inference Speed And Reduce Costs In Production

How to optimize LLM inference speed and reduce costs in production Announcing our Series F . Learn more

Model performance

How to optimize LLM inference speed and reduce costs in production

Cut LLM inference latency and cost with continuous batching, speculative decoding, quantization, and more. Learn which techniques fit your workload.

Authors

Chloe Florit

Last updated July 23, 2026

Share

TL;DR LLM inference costs too much and responds too slowly when GPUs sit idle, repeat work, and move too much data. This post covers some of the most popular techniques to optimize inference: continuous batching, speculative decoding, KV cache reuse, quantization, smarter routing, and more.

Your model isn’t slow. Your inference is. A properly optimized model will outperform a frontier model that isn’t. LLM inference has two main phases: prefill, when the model processes the prompt, and decode, when it generates the response one token at a time. In this post, we’ll cover techniques used to optimize both phases to improve latency, throughput, and cost when running AI models in production . 1. Continuous batching A batch is a group of requests processed together on the GPU at the same time. Batching matters because GPUs are built to handle multiple computations from different requests in parallel. During one decode iteration, the GPU generates one token for every active request in the batch. The problem with traditional batching is that the server waits for every request in the batch to finish before accepting new ones. You can’t add requests mid-batch. Continuous batching fixes this. At each iteration, the GPU processes all active requests together. When a new request arrives, the GPU generates new tokens for each request with no wait time. Iterations 1-20: GPU processes requests [A,B]. Each request gets one token per iteration.

Iteration 21: Request C joins. GPU processes [A, B, C].

The result is faster response times and reduced queuing latency. 2. Speculative decoding During the decode step, tokens are typically generated one by one. Speculative decoding lets you generate multiple candidate tokens in advance with a smaller model and then verifies them all in a single pass of the main, larger model. Verification is cheaper than generation because the target model can verify all of the draft tokens in parallel. If the target model generated those same tokens itself, it would have to do so sequentially, one token at a time. If most of the draft tokens are accepted, the system produces multiple tokens for a single target-model pass, which reduces latency and increases throughput (tokens per second). Here are the main speculative decoding techniques: Draft-target speculative decoding Draft-target speculative decoding is when a smaller draft model “guesses” several tokens ahead and a full target model then verifies those guesses. Correct guesses are accepted in order. When the target model disagrees with a guess, it replaces that token with its own choice, discards the remaining draft tokens, and the draft model starts guessing again from there. ✕ Draft-target speculative decoding Medusa Medusa adds several prediction heads on top of the LLM. Each head independently predicts a future token position, such as token +1, token +2, or token +3. Then the full transformer verifies all guesses at once. The added heads are lightweight because they are simple linear projections: they take the model’s hidden state (the model’s internal representation of the latest token and the context before it) and apply a learned matrix multiplication to produce token predictions. They don’t include the attention, memory, or multi-step reasoning of full transformer layers. EAGLE-3 EAGLE-3 is a small 1-2 layer transformer that you add on top of the LLM. It cheaply drafts several tokens ahead. Unlike Medusa, EAGLE’s small transformers generate tokens one at a time, so its drafts are more coherent and accurate.  EAGLE-3 is still much cheaper than running the full model because its draft module has far fewer layers than the target LLM. N-gram speculation N-gram speculation is a lightweight speculative decoding technique that speeds up generation by reusing repeated patterns from the prompt or prior output. Instead of using a separate draft model, it checks whether the tokens being generated match a sequence that appeared earlier. If so, it proposes the tokens that followed that earlier sequence. The target model verifies the guess, keeps the matching tokens, and discards the rest. It works best for repetitive or structured outputs like code, JSON, templates, or repeated phrasing, and is less useful for open-ended reasoning or explanations. Speculative decoding works best at low batch sizes because there's spare compute. At large batch sizes, the GPU is already busy serving all the requests, so there's no spare compute (tensor cores) and speculation would have to fight for that same compute (to run the draft model and verify multiple tokens per request), which can hurt throughput. 3. KV cache optimizations “Keys” help the model figure out which words to pay attention to, and “values” determine what information gets added to a word's meaning based on the context. Together, they are cached as the “KV cache”. KV cache-aware routing :
 automatically routes requests to replicas* that have the necessary context cached, which eliminates redundant prefill compute. This is especially useful for shared prompts, like system messages or few-shot examples. NVIDIA Dynamo uses this approach natively.

CPU offloading : moves unused cache blocks to CPU RAM to reduce GPU memory pressure.

Note: A replica is an instance (copy) of a model running on one or more GPUs. 4. Quantization Quantization makes a model smaller and faster by storing its weights in a lower-precision format, which can help reduce LLM inference cost for memory-bound workloads. Model weights are stored in VRAM (high-bandwidth memory attached to the GPU). During inference, the GPU has to move those weights from VRAM into its compute units (e.g., registers or shared memory) to perform calculations. If you cut the weight size, you cut the bytes you must fetch by ~2x or ~4x. For example, FP16 uses 2 bytes per weight, while FP8 uses 1 byte, and FP4 uses 0.5 bytes. A sign, mantissa, and exponent make up a floating-point number. The sign says if the number is positive or negative, the mantissa holds the actual numbers in binary, and the exponent moves the...