Mixture Of Experts Models Get More From Speculative Decoding

Why MoE models get more from speculative decoding North Mini Code. Cohere's first model for developers. Learn more

Apr 21, 2026

9 minute read

Why MoE models get more from speculative decoding

Background Large Language Models generate text one token at a time. Each token requires a full forward pass through the model, making generation inherently slow. Speculative decoding (SD) attacks this bottleneck: a small, fast “draft” model guesses the next tokens, and the full “target” model verifies all K + 1 K+1 K + 1 of them in a single forward pass. When the guesses are right, you get multiple tokens for nearly the cost of one.

Mixture-of-Experts (MoE) models add an interesting twist. Each token only activates a small subset of the model’s parameters — called “experts” — making individual tokens cheap. But during verification, K + 1 K+1 K + 1 tokens may each route to different experts, potentially loading far more weights from memory than a single-token decode would. Does this extra loading erase the speedup that speculative decoding is supposed to deliver?

Recent work by MoESD predicts that the answer is nuanced: MoE’s low arithmetic intensity creates a non-monotonic speedup curve, where gains first increase with batch size before declining.

In this post, we validate this prediction on our production MoE models and extend the analysis in two directions. First, we examine temporal correlation in expert routing — a structural property of MoE models documented in prior work — and quantify its impact on SD verification cost. Second, we uncover a distinct mechanism behind the surprisingly high speedup at BS=1, where fixed-overhead amortization plays a role that routing analysis alone cannot explain. Finally, we also draw out concrete design implications for co-optimizing model sparsity and speculative decoding. Observing the non-monotonic speedup curve In LLM inference, the inference system batches multiple concurrent requests so that their decode steps execute together in a single forward pass — the number of requests processed in parallel is the batch size (BS). Because LLM decode is memory-bandwidth-bound at small BS and becomes compute-bound at large BS, the batch size fundamentally shapes how much benefit speculative decoding can deliver. Using vLLM, we benchmarked SD speedup across batch sizes for both a dense and a MoE model.

Dense baseline ( Command A , 111B). Speedup vs. batch size shows monotonic decrease — gains are highest at low BS where the model is bandwidth-bound.

Figure 1: Command A speedup Cohere MoE + SD K = 3 K=3 K = 3 . Speedup first increases then declines — the non-monotonic shape predicted by MoESD . The gain at BS=1 is unusually high; we revisit this anomaly in a later section.

Figure 2: Cohere MoE speedup The contrast is striking: where the dense model’s SD gains monotonically decay, the MoE model exhibits a sweet spot at moderate batch sizes. What creates this sweet spot, and why does BS=1 stand out? Why MoE creates a sweet spot: arithmetic intensity The key to the non-monotonic curve lies in arithmetic intensity. For a MoE layer with N total experts, where each token selects k k k of them (top-k routing), processing T T T tokens with S S S shared experts gives an arithmetic intensity of: A I = T ( k + S ) N + S AI = \frac{T(k+S)}{N+S} A I = N + S T ( k + S ) ​

When S = 0 : A I = T k / N S=0: AI=Tk/N S = 0 : A I = T k / N . Sparsity (low k / N k/N k / N ) keeps A I AI A I low, pushing the layer deeper into the bandwidth-bound regime at any given T T T .

This creates three batch-size regimes for SD, as analyzed in MoESD :

Low BS — partial expert loading. The model is memory-bandwidth-bound, but decode doesn’t yet load all experts. Verification adds extra expert weight loading on top of decode cost, limiting SD gains.

Moderate BS — the sweet spot. Both decode and verification load nearly all experts anyway — verification adds almost no extra weight loading. Because A I AI A I remains low (thanks to sparsity), the model stays bandwidth-bound, making verification tokens nearly free. SD gains peak here.

High BS — compute-bound. Arithmetic intensity exceeds the machine’s ops-to-byte ratio. Every extra verification token costs proportional compute, and SD gains vanish.

Sparser models (lower k / N k/N k / N ) stay bandwidth-bound longer, pushing the sweet spot to higher batch sizes. Connecting to SD speedup Following MagicDec and MoESD , the SD speedup is dominated by the verification cost ratio T t ( K + 1 ) / T t ( 1 ) T_t(K + 1)/T_t(1) T t ​ ( K + 1 ) / T t ​ ( 1 ) . Draft overhead is typically negligible. For K = 3 K=3 K = 3 , the key metric is: T t ( 1 ) T t ( 4 ) ∈ ( 0 , 1 ] \frac{T_t(1)}{T_t(4)} \in (0,1] T t ​ ( 4 ) T t ​ ( 1 ) ​ ∈ ( 0 , 1 ]

As this approaches 1 (verification ≈ free), speedup approaches the theoretical maximum of A L AL A L (acceptance length). The remaining sections measure and model this ratio via expert overlap analysis and Amdahl’s Law . Co-designing model sparsity and speculation The three-regime analysis yields a concrete design principle: for a given target batch size, the sparsity ratio k / N k/N k / N determines whether the model operates in the bandwidth-bound sweet spot or crosses into compute-bound territory where SD gains vanish.

Sparser models (lower k / N k/N k / N ) stay bandwidth-bound to higher batch sizes, widening the sweet spot. Denser models hit the compute-bound regime earlier. This means sparsity is not just a parameter-efficiency knob — it directly controls how much benefit speculative decoding can deliver at a given target BS load.

The shared-to-routed expert ratio adds a second lever: shared experts reduce verification cost at low BS (by lowering the fraction of forward-pass time spent on routed expert weight loading), but raise the effective k / N k/N k / N , pushing the compute-bound transition earlier. In the limit, a model that shares all experts is equivalent to a dense model, with no sweet spot at all.

For systems where the target BS is known at design time, these two knobs can be set accordingly:

High target BS — maximize SD benefit by: Lowering k / N k/N k / N : fewer active experts per token keeps arithmetic intensity low and the model bandwidth-bound even at large batch sizes.

Increasing the routed-to-shared expert ratio : more routed (fewer shared) experts keep the effective k / N k/N k / N low, preserving the bandwidth-bound regime.

Low target BS — the calculus shifts: shared...