novitalabs/chord

novitalabs/chord

Description: W4A16 MoE GEMM CUDA kernel. Tuned for H200, B200 & B300.

Language: Python

License: Apache-2.0

Stars: 0

Forks: 0

Open issues: 0

Created: 2026-08-03T15:57:09Z

Pushed: 2026-08-04T06:25:36Z

Default branch: main

Fork: no

Archived: no

README:

Novita Labs' production MoE CUDA kernel

| Documentation | Novita AI | Blog |

---

The chord repository publishes the Python package chord_kernels: Novita Labs' in-house W4A16 MoE CUDA operator — BF16 activation, INT4 weight (stored as unsigned nibbles, decoded as code - 8), group-32 scale — through the indexed interface, plus a thin layer adapter for inference framework integration. The kernel runs in Novita's production inference service; the CUDA template directory keeps only the minimal dependency closure it needs at runtime.

The operator is open-sourced interface by interface. This release publishes the indexed interface, which fits single-node deployments: its routing metadata (sorted ids, expert ids) addresses one node's local experts directly. See [Roadmap](#roadmap) for what follows.

Where the name comes from

In music theory a chord is two or more notes of different pitch sounded together. The name points at how this operator runs: a gathered BF16 activation, an INT4 weight and a group-32 scale are struck at once in one fused GEMM, each keeping its own pitch.

Supported configurations

| GPU | Compute capability | Scenario | Profile | | --- | --- | --- | --- | | Hopper (H200) | SM90 (9.0) | Prefill, EP8 | h200_prefill_ep8 | | Hopper (H200) | SM90 (9.0) | Decode, EP8 | h200_decode_ep8 | | Blackwell (B200/B300) | B200: SM100 (10.0); B300: SM103 (10.3) | Decode, EP8 | blackwell_decode_ep8 |

All three parts are compiled, run and performance-verified.

Each profile fixes a tensor-core instruction family and the physical weight layout that goes with it, so the profile must be chosen when the weight is packed and cannot be switched at runtime; a mismatched layout is rejected before launch. A production deployment prepares the matching profile per P/D instance, or keeps two packed copies of the weight. The scheduling details (WGMMA vs MMA, swap-AB, block-M model, stream-K) are in [docs/tuning.md](docs/tuning.md).

tests/test_w4a16.py checks every tuning row against a plain-PyTorch reference on the running device before timing it, so a deployment can reproduce both the correctness and the performance numbers for its own GPU with one command.

blackwell_decode_ep8 is the Blackwell decode profile name. B200/SM100 and B300/SM103 share one schedule, but the JIT targets sm_100a and sm_103a separately per actual compute capability and does not reuse a cubin across them.

Measured performance

humming is the public Humming indexed path, chord is this repository's profile for that scenario; both are timed on the same GPU at the same shape and the same routing draw. Times are per-call microseconds, lower is better. gate_up + down is the speedup of the two stages summed, which is what one MoE layer actually pays.

*Per-call latency from the tables below; each panel annotates the gate_up + down layer speedup range. Regenerate after re-measuring with python docs/assets/benchmark_chart.py.*

H200 EP8 prefill (h200_prefill_ep8)

| Stage | num_tokens_total | humming µs | humming TFLOPS | chord µs | chord TFLOPS | Speedup | gate_up + down | | --- | --- | --- | --- | --- | --- | --- | --- | | gate_up | 1024 | 375.0 | 160.36 | 313.5 | 192 | 1.20 | 1.20 | | gate_up | 2048 | 466.3 | 257.92 | 383.4 | 314 | 1.22 | 1.19 | | gate_up | 4096 | 608.4 | 395.34 | 545.8 | 441 | 1.11 | 1.13 | | gate_up | 8196 | 1085.7 | 443.08 | 997.7 | 482 | 1.09 | 1.11 | | gate_up | 16384 | 1903.0 | 505.55 | 1743.1 | 552 | 1.09 | 1.11 | | down | 1024 | 195.9 | 153.43 | 162.9 | 185 | 1.20 | | | down | 2048 | 235.1 | 255.81 | 204.5 | 294 | 1.15 | | | down | 4096 | 337.4 | 356.40 | 293.2 | 410 | 1.15 | | | down | 8196 | 606.4 | 396.64 | 533.4 | 451 | 1.14 | | | down | 16384 | 1059.4 | 454.07 | 937.3 | 513 | 1.13 | |

H200 EP8 decode (h200_decode_ep8)

Tokens per GPU is bs * (mtp + 1).

| Stage | tokens/GPU | humming µs | humming GB/s | chord µs | chord GB/s | Speedup | gate_up + down | | --- | --- | --- | --- | --- | --- | --- | --- | | gate_up | 20 | 267.6 | 2845.08 | 221.9 | 3431 | 1.21 | 1.24 | | gate_up | 30 | 281.3 | 2826.64 | 245.6 | 3237 | 1.15 | 1.20 | | gate_up | 40 | 284.0 | 2802.95 | 253.7 | 3137 | 1.12 | 1.16 | | gate_up | 50 | 296.4 | 2687.97 | 259.9 | 3066 | 1.14 | 1.17 | | down | 20 | 146.2 | 2618.17 | 111.9 | 3421 | 1.31 | | | down | 30 | 152.2 | 2633.29 | 116.8 | 3432 | 1.30 | | | down | 40 | 153.3 | 2624.30 | 123.0 | 3271 | 1.25 | | | down | 50 | 156.3 | 2582.34 | 126.4 | 3195 | 1.24 | |

B300 EP8 decode (blackwell_decode_ep8)

| Stage | tokens/GPU | humming µs | humming GB/s | chord µs | chord GB/s | Speedup | gate_up + down | | --- | --- | --- | --- | --- | --- | --- | --- | | gate_up | 20 | 319.1 | 2385.78 | 146.0 | 5215 | 2.19 | 2.15 | | gate_up | 30 | 320.4 | 2481.51 | 162.3 | 4900 | 1.97 | 1.98 | | gate_up | 40 | 320.6 | 2482.68 | 175.2 | 4544 | 1.83 | 1.88 | | gate_up | 50 | 320.9 | 2482.44 | 182.9 | 4355 | 1.75 | 1.81 | | down | 20 | 174.8 | 2190.34 | 83.8 | 4565 | 2.09 | | | down | 30 | 181.3 | 2211.10 | 91.3 | 4391 | 1.99 | | | down | 40 | 181.4 | 2217.48 | 91.9 | 4379 | 1.97 | | | down | 50 | 181.7 | 2221.51 | 95.1 | 4245 | 1.91 | |

The Blackwell speedups are larger than the H200 ones mostly because of the baseline, not because the Blackwell schedule is better tuned than the Hopper one. Public Humming has no SM100/SM103 heuristics and falls back to its SM80 rules there, so its B300 numbers are an untuned reference point — note how its time barely moves from 20 to 50 tokens per GPU, and how it reads roughly the same GB/s on B300 as on H200 despite the wider memory system. Read the H200 ratios as the honest tuned-to-tuned comparison.

Requirements and installation

Requires Linux x86_64, Python 3.10+, a compatible PyTorch 2.1+, a CUDA Toolkit (NVRTC and CUDA headers), a host C++ compiler, and a target GPU. Installation builds only the Python wheel and compiles no Torch/CUDA extension; the launcher, repack kernel and indexed kernel are JIT-compiled and cached on first use. An ordinary isolated build therefore does not pull a second copy of Torch and does not need --no-build-isolation:

python -m pip install -v -e ".[test]"

To build...