togethercomputer/xorl-wheels sgl_fused_moe_trtllm_sm100_fi0618_cu130_nvcc131_sm100a_sm103a_1b699bf

FlashInfer 0.6.18 SM100a+SM103a MoE-LoRA AOT — Kimi-K3 SiTU, gate_up GEMM hang fix

Repository: togethercomputer/xorl-wheels

Tag: sgl_fused_moe_trtllm_sm100_fi0618_cu130_nvcc131_sm100a_sm103a_1b699bf

Published: 2026-10-06T03:22:17Z

Prerelease: yes

Release notes: This prerelease contains the AOT sgl_fused_moe_trtllm_sm100 for FlashInfer 0.6.18, built for sm100a (B200) and sm103a (B300). It replaces sgl_fused_moe_trtllm_sm100_fi0618_cu130_nvcc131_sm100a_875b7d7.

NVFP4 multi-LoRA passed 28/28 checks on B200. On B300, Kimi-K3-NVFP4 MoE LoRA (SiTU) serves correctly. Both runs loaded this AOT binary; nothing was JIT-compiled.

Changes since 875b7d7

The kernel source is tgl PR #1243 (togethercomputer/tgl, commit 555eb9f):

• SiTU activation for the decomposed NVFP4 MoE-LoRA op (Kimi-K3).
• Per-expert beta and linear beta are read from gemm1_alpha / gemm1_beta.
• The up half is rescaled by g1_alphas_up / g1_alphas before the tanh.
• The op runs without LoRA when given empty bridges.
• Empty gate_up_lora_delta / activation_lora_input run the same pipeline with no delta.
• SiTU batches that carry no adapter use this path. trtllm-gen ships no fused SiTU GEMM1 with per-token NVFP4 activation scaling.
• Hang fix.
• The gate_up GEMM (NVFP4 and bf16 launchers) chose its config and workspace for swapped (N, K).
• It then hung the GPU whenever hidden % 1024 != 0 (Kimi-K3's 3584 latent experts). SwiGLU is affected as well.
• Activation enum synced with flashinfer 0.6.18. The overlay ActivationType now has Situ = 10, and Identity moves from 7 to 9.

Install and architecture

pip install --no-deps --require-hashes -r <(echo "https://github.com/togethercomputer/xorl-wheels/releases/download/sgl_fused_moe_trtllm_sm100_fi0618_cu130_nvcc131_sm100a_sm103a_1b699bf/xorl_sgl_trtllm_aot-0.6.18+csrc1b699bf.cu131-py3-none-linux_x86_64.whl --hash=sha256:c62d82540437179d7824a1b164da94a20cb938e534c6ce5b88ecbd40a1daf04e")

FlashInfer's `is_aot` only checks that the `.so` exists. It does not check the GPU architecture. The previous sm100a-only wheel would therefore also have been loaded on B300. This build carries cubins for both architectures (10 sm_100a + 10 sm_103a).

Build

• Source commit: 555eb9f7a72d1d09d96cac2a87e82c7f7931f147
• Kernel source tree: 1b699bf5ad5977353b2e56b2c06f47165991667b
• Tree path: python/sglang/kernels/ops/moe/trtllm_lora_temp/data
• FlashInfer Python and cubin: 0.6.18
• FlashInfer JIT cache: 0.6.18+cu130
• Torch: 2.13.0+cu130
• CUDA compiler: 13.1.115
• Host compiler: gcc 11.4 (Ubuntu 22.04)
• TVM FFI: 0.1.11
• GPU architecture: sm100a + sm103a (FLASHINFER_CUDA_ARCH_LIST="10.0a 10.3a")
• Minimum host libraries: GLIBC 2.34, GLIBCXX 3.4.29, the same as 875b7d7. A build made inside an Ubuntu 24.04 image needs GLIBCXX 3.4.32 and fails to load on 22.04 hosts. METADATA.json now records both minimums.

Rebuild when the kernel source tree or the FlashInfer version changes. The AOT lookup silently prefers the installed .so over a new JIT build, and pip cannot enforce the source-tree match.

Validation

B200: SwiGLU regression, the same harness as 875b7d7

• Model: NVFP4/Qwen3-30B-A3B-Instruct-2507-FP4
• Hardware: four B200 GPUs, TP=4
• Adapters: eight rank-16 shared-outer adapters
• Backends: experimental_sgl_trtllm, virtual experts, LoRA backend triton
• Harness: test/manual/lora/test_moe_lora_trtllm_correctness.py

| Phase | Passed | | --- | --- | | Individual adapters | 8/8 | | Eight adapters in one batch | 8/8 | | Mixed adapter and base requests | 12/12 | | Total | 28/28 |

The run used a fresh FlashInfer workspace, and no sgl_fused_moe_trtllm_sm100 JIT build appeared, so the AOT binary served it. With FLASHINFER_DISABLE_JIT=1, the artifact loaded with is_aot=True in 0.32 s.

B200 and B300: op-level numerics

test/registered/lora/test_trtllm_fp4_situ_moe_lora.py passed 2/2 on both GPUs with FLASHINFER_DISABLE_JIT=1.

• Setup: SiTU, with and without a LoRA delta, compared against a dequantized bf16 reference at Kimi-K3 dimensions (H=3584, I=384).
• Result: cosine 0.986. The trusted fused SwiGLU cubin reaches 0.987 on the same harness.

B300: Kimi-K3 end to end

• Model: nvidia/Kimi-K3-NVFP4
• Hardware: eight B300 GPUs, TP=8
• Backends: experimental_sgl_trtllm, virtual experts, shared-outer experts
• LoRA targets: q_b_proj kv_b_proj gate_up_proj down_proj
• Adapters: synthetic rank-16

| Check | Result | | --- | --- | | Base output coherent, and unchanged across adapter load/unload | pass (bit-identical) | | Zero-B routed-expert and shared-expert adapters equal base | pass (bit-identical) | | Random routed-expert, shared-expert and MLA adapters change the output and stay coherent | pass | | Same output as the JIT build of the same source | pass (bit-identical base) | | AOT load (FLASHINFER_DISABLE_JIT=1) | is_aot=True, 0.20 s |

Known: some outputs vary across server launches. Requests with an active adapter can differ between separate server launches, up to ~0.4 in prompt logprob on a few tokens. Within one launch, results are deterministic, and base-only requests did not vary. The cause has not been isolated, and it does not depend on this binary.

Performance has not been benchmarked. Under LoRA, the SiTU path uses the scalar activation kernel and the unfused activation quantization.