Hardware Aware Dynamic Speculative Decoding

Hardware-Aware, Dynamic Speculative Decoding (DSD) | Cohere

Jul 10, 2026

8 minute read

Hardware-aware dynamic speculative decoding

Speculative decoding (SD) is a popular technique for accelerating large language model (LLM) inference without any loss in quality. LLM inference generates one token at a time. SD instead uses a smaller draft model to propose multiple tokens, which are then verified by the large target model in a single timestep. Because more than one token can be accepted per step, this speeds up generation.

SD achieves its speedup by exploiting the tradeoff between the compute and memory bandwidth of a GPU. Batching multiple requests improves GPU utilization, but the operating regime depends on the batch size (BS): At small BS , inference is memory bandwidth-bound : the cost of loading LLM weights from HBM into SRAM dominates. At large BS , inference becomes compute-bound : the matrix multiplications of LLM weights against many request tokens dominate.

The time spent in SD has two main components:

Time in SD = Time to draft tokens (small draft model) + Time to verify draft tokens (large target model)

Draft models are much cheaper to run. If the draft produces K tokens, the target model must process BS × (K+1) tokens, which makes verification the bottleneck since it does K+1 times more computation for a given BS. On modern GPUs, the compute units are typically about two orders of magnitude faster than memory bandwidth, so during memory bandwidth-bound, small-BS inference, the compute units sit largely idle. SD leverages this gap by pushing K+1 times more tokens through the target model during verification, using compute that would otherwise be wasted. Challenges in speculative decoding Dynamic batching in production As batch size grows, ordinary LLM inference becomes increasingly compute-bound, leaving no idle compute for SD to exploit. In this regime, SD can actually be slower than ordinary inference. This limits SD in production systems, where BS is rarely small and changes dynamically. Reinforcement learning (RL) RL alternates between a training phase, which updates model weights, and a rollout phase, which generates data and reward signals to train on. The rollout phase is the main bottleneck and can consume up to 85% of resources . Since scaling RL is an important avenue for improving model intelligence, accelerating RL rollout inference is an important problem to solve.

However, RL with reasoning models produces a long-tail distribution: a single request in a batch can keep generating for a very long time, stalling the rest of the batch and wasting resources. SD would help with these long-tail generations, but because it hurts throughput at high BS, its overall utility in a full deployment is limited.

Together, these challenges point to the same requirement: a better SD system should control the optimal K based on hardware constraints , so that it remains useful in full production systems and large-scale RL rollouts. Dynamic speculative decoding This is where hardware-aware dynamic SD (DSD) comes in. DSD improves on standard SD by making the number of draft tokens adaptive. In SD, K is fixed; in DSD, the optimal K is chosen based on the interaction between the model and the hardware. DSD increases K when inference is memory bandwidth-bound and decreases it when inference is compute-bound. For dense models, this generally means a higher K at low BS and a lower K at high BS, with the optimal K decreasing monotonically as BS grows. For MoE (Mixture-of-Experts) models, the optimal K is non-monotonic in BS: At low BS , optimal K starts low because verification loads additional experts. At mid BS , optimal K increases because almost all experts are already loaded, so verification adds little extra expert loading. At high BS , optimal K decreases again, just like a dense model, because the computation is now compute-bound.

For a deeper treatment of these mechanics, see our previous blog post on the interaction of SD with MoEs.

So, how do we find the optimal number of draft tokens? We need to understand the marginal contribution of going from K to K+1 draft tokens. The computational cost of processing one draft token is the same regardless of its position — the first and last draft tokens cost the same. Its contribution to the Acceptance Length (AL) , however, depends heavily on position: it decays exponentially , so earlier tokens are far more likely to be accepted than later ones. We therefore need a metric that captures this tradeoff when adding one more draft token. Existing DSD work, such as TurboSpec , uses goodput as this metric. We use goodput too, but simplify it to:

goodput = AL / ITL

AL is the acceptance length, and ITL is the inter-token latency (largely the sum of the draft and verify times). This captures the tradeoff between the marginal contribution of K draft tokens to AL and the wall clock cost, with a higher value implying a better speedup. This simple formulation makes DSD easy to adapt to future model architectures, and since ITL already encapsulates the combined impact of all operations, there's no need to profile each component individually and recombine them. We run offline profiling to measure AL and ITL, find the optimal K, and store it as a lookup table used at runtime. This offline table solves the cold-start problem and is extensible: it can incorporate live AL and ITL statistics from the engine's runtime metrics, allowing DSD to adapt to changing workloads. Results We compare three configurations: vanilla (no speculative decoding), fixed-K SD (our SD baseline, implemented with an EAGLE draft head at a fixed K=3), and DSD (dynamic K) on the MT-Bench dataset with the number of samples upsampled to 20 * BS, which implies 20 waves of BS being used.

We profiled the optimal K selected by DSD for both Command A (Dense) and Command A+ (MoE). The empirical results validate the trends described above: for the dense model, the optimal K decreases monotonically as BS increases, whereas for the MoE model it does not — Command A+ can reach a higher optimal K at mid BS. We also benchmark TOPS/user (token output per second, per user). For Command A (Dense) , the gains from DSD are clear across the whole BS range: At low BS , DSD matches the speedup of SD (K=3). At high BS (64/128) , DSD is faster than both SD and the vanilla model. At very high BS (256) , DSD matches vanilla TOPS — even though SD regresses at BS 128 and 256....