When Faster Not Identical Moe Numerics

Training-Inference Parity in MoE Models: Where Numerics Drift

GLM 5.2 is live! Opus-level intelligence at open-source rates. Pay per token on serverless. Try it today.

Blog

Training-Inference Parity in MoE Models: Where Numerics Drift Training-Inference Parity in MoE Models: Where Numerics Drift Kernel fusions that are mathematically equivalent can still drift numerically. This draft walks through the parity bugs we hit across Kimi K2.5 serving and Qwen3.5-MoE training, plus the fixes that kept k3 near the noise floor.

PUBLISHED 3/10/2026

TRAINING REFERENCE FUSED INFERENCE SAME WEIGHTS + SAME INPUT, BUT A DIFFERENT REDUCTION ORDER TP partial outputs NCCL ring reduce-scatter Standalone RMSNorm Reference logprobs PER-CHUNK OWNERS GPU0 owns chunk 0 GPU1 owns chunk 1 NCCL ORDER ROTATES chunk 0: r0 + r3 + r2 + r1 chunk 1: r1 + r0 + r3 + r2 LOCAL BUFFERS buffer 0 buffer 1 buffer 2 LAMPORT LOCAL ORDER chunk 0: r0 + r1 + r2 + r3 chunk 1: r0 + r1 + r2 + r3 Scatter-write HBM buffers Local fixed-order sum on every GPU Fused RMSNorm + residual Served logprobs K3 DRIFT COMPOUNDS In an LLM, KL compares the full next-token probability vector from training and inference. The top token can stay the same, but probability moving across lower-ranked tokens still makes KL grow. KL MEASURES DISTRIBUTION MISMATCH, NOT JUST TOP-1 AGREEMENT EXAMPLE TOKEN PROBABILITIES same prompt, same decode position training inference larger shift means larger KL token probability 0.0 0.1 0.2 0.3 0.4 top token runner-up alt token A alt token B training and inference can still agree on the top token, while lower-ranked probabilities drift apart

On this page

When Faster ≠ Identical: Numerical Pitfalls in Serving MoE Models

Kernel fusions that are mathematically equivalent can still drift numerically. Here are the parity bugs we hit across both Kimi K2.5 serving and Qwen3.5-MoE training bring-up.

Why This Matters

When you train a model and serve it for inference, you expect them to agree. The same weights, the same input, the same output distribution. This training–inference numerical parity matters more than it sounds:

RLHF / GRPO reward integrity. The reference model's logprobs anchor the KL penalty. If inference produces different logprobs than training for the same weights, the policy can exploit the gap without actually improving.

Reproducibility. Numerical drift from kernel fusions is invisible — weights are identical, architecture looks the same, yet outputs diverge.

Customer trust. Users fine-tune on our platform and expect the served result to match what training optimized for.

For dense models, parity is relatively easy. Mixture-of-Experts models like Kimi K2.5, Qwen3.5-MoE, and DeepSeek V3 are harder. With routed experts, shared expert pathways, and all-reduce communication twice per layer across deep stacks, there are many places where "mathematically equivalent" optimizations produce numerically different results.

This post catalogs the pitfalls we found. Each is a class of optimization that inference engines use for performance, but that can silently break numerical alignment. We found most of these while bringing up Kimi K2.5 on our serving stack, then saw the same failure mode again while debugging Qwen3.5-MoE. We will use FlashInfer and TRT-LLM style fused kernels as concrete examples.

The Fundamental Issue: FP Addition Is Not Associative

Every pitfall below reduces to one fact: floating-point addition is not associative. Even in FP32:

(a + b) + c ≠ a + (b + c)

Each addition rounds the result to the nearest representable value. Different orderings produce different intermediate values, which get different rounding errors. The errors are tiny per operation, but they compound through 61 transformer layers — and MoE routing amplifies them (a small change in hidden state can flip which experts get selected, cascading through the rest of the network).

Pitfall 1: All-Reduce Topology Differences

What it is

In tensor-parallel inference, every linear layer's output must be summed across GPUs via all-reduce . This happens twice per layer: after the attention output projection, and after the MLP/MoE.

Training typically uses NCCL , which implements all-reduce as a reduce-scatter followed by an all-gather . In the reduce-scatter phase with a ring topology, the data is divided into chunks — one per GPU. Each chunk is accumulated as partial sums flow around the ring, starting from the GPU that "owns" that chunk. For 8 GPUs, this means different parts of the hidden vector see different summation orders:

NCCL ring reduce-scatter (8 GPUs): chunk 0 (owned by GPU0): r0 + r7 + r6 + r5 + r4 + r3 + r2 + r1 chunk 1 (owned by GPU1): r1 + r0 + r7 + r6 + r5 + r4 + r3 + r2 chunk 2 (owned by GPU2): r2 + r1 + r0 + r7 + r6 + r5 + r4 + r3 ...each chunk starts from its owner, accumulates around the ring

NCCL RING REDUCE-SCATTER LAMPORT LOCAL SUM CHUNK OWNERSHIP ROTATES ON THE LEFT; LOCAL ORDER STAYS FIXED ON THE RIGHT ONE EXAMPLE CHUNK: SAME VALUES, DIFFERENT FP32 ADDITION ORDER training / NCCL reference chunk ownership rotates with the ring R0 R3 R2 R1 accumulated tensor R0 ring order for this chunk R0 -> R3 -> R2 -> R1 same chunk, rotating owner: NCCL follows the ring, then reduces inference / Lamport local sum every GPU applies one fixed local order R0 R0 R1 R2 R3 accumulated tensor FP32 accumulation order R0 -> R1 -> R2 -> R3 every GPU keeps one fixed local order ROUNDING ORDER CHANGES Both kernels compute the same exact sum in real arithmetic. But fp32 addition is order-sensitive, so rotating the owner changes which partial sums round first even when the exact sum is identical. SAME SUM, DIFFERENT FP32 ACCUMULATION ORDER All-Reduce Topology NCCL rotates chunk ownership around the ring, while Lamport kernels sum in a uniform local order. Exact arithmetic agrees; floating-point accumulation does not. Inference serving engines often replace NCCL with custom all-reduce kernels for lower latency. FlashInfer's Lamport IPC kernel (derived from TRT-LLM) uses a different approach: each GPU writes its data to all other GPUs' buffers via CUDA IPC, then every GPU reads all contributions locally and sums them in a fixed order:

Lamport kernel (all elements, on every GPU): every chunk: r0 + r1 + r2 + r3 + r4 + r5 + r6 + r7

Both accumulate in FP32. Both produce the correct sum in exact arithmetic. But the per-chunk rotation in NCCL means different elements of the hidden vector see different addition orders, while the...