NVIDIA/TensorRT-LLM v1.4.0rc1

v1.4.0rc1

Repository: NVIDIA/TensorRT-LLM

Tag: v1.4.0rc1

Published: 2026-10-22T01:38:13Z

Prerelease: yes

Release notes:

Highlights

• Known Issues
• Disaggregated serving with the default Python KV cache transceiver may fail generation requests with a KV cache transfer error after streaming requests are cancelled (observed on A100).
• The OpenAI JSON-schema serving example (examples/serve/openai_completion_client_json_schema.py) may return an unbounded response with Qwen3 models.

• Model Support
• Add DeepSeek-V4.1-Flash support #19727
• Support NVFP4 GLM-5.3-Flash checkpoints #20127
• Add Qwen3.8-Flash-Next NVFP4 as an experimental Modeling V2 target #20143
• Support LiLiCorr draft checkpoints in DFlash #19856
• Remove the automatic W4A16 fallback for Nemotron-H NVFP4 checkpoints on Hopper #15772

• API
• Default to KV cache manager V2 and the Python KV cache transceiver (BREAKING) #17495
• Remove unused LlmRequest fields (BREAKING) #19967
• Replace the C++ ArrayView API with std::span #20205
• Select Modeling V2 through LLM configuration and arguments #20077
• Publish serving configuration schemas #19906
• Add a KV cache lending API for transfer backends #19722
• Expose native KV event streaming and routing load #19440
• Support runtime sleep and wakeup through LLM and trtllm-serve, including MiniMax-H3 #19016, #19604
• Unify compiled-artifact cache configuration through TRTLLM_CACHE_DIR #18897
• Make KV cache V2 pool rebalancing parameters configurable #19077
• Reorganize VisualGen attention modules, keeping deprecated import paths and lazy exports #20214, #20215, #20218

• Feature
• Integrate the Mooncake connector with KV cache manager V2 #19171
• Fill the KV cache V2 host tier progressively #19650
• Add RoPE precision options for NVFP4 cold-page compression #19391
• Honor per-request seeds in batched TorchSampler and one-model speculative acceptance #19719, #20147
• Add ragged DSpark verification with confidence-based policy and calibrated scoring #19707, #19818, #20071
• Add deterministic FP32 GVR V2 top-K decoding and reduce its overflow overhead #20100, #20290
• Add an opt-in setting to compile only piecewise CUDA graph forwards #18366
• Support Rubin locality-domain execution and extend SM107 and multi-node MegaMoE initialization #19272, #19183
• Optimize Kimi K3 on Rubin and avoid cuBLAS fallback in KDA prefill #19960, #20084
• Support packed query heads and compact inputs in Blackwell CuTe DSL MLA #19338
• Add PrimTS dense BF16, mixed BF16/FP8, and FP8 context attention paths #20006
• Use paged attention for MiniMax-M3 dense layers and cut NVFP4 KV host overhead #20159, #19974
• Limit GLM-5.3-Flash indexer scoring to visible pools with BF16 tensor cores #20251
• Add token-sharded tensor parallelism for Wan in VisualGen #19713
• Support MiniMax-H3 parallel tiled VAE decoding and Ulysses parallelism #19506
• Speed up MiniMax-H3 video-VAE decoding and fuse AdaLN RMSNorm modulation #19774, #19772
• Unify multimodal encoder caching and reduce encoder output copies #17876, #19799
• Optimize Qwen vision encoders and decode JPEGs through torchvision #19797, #19794
• Reduce per-response overhead in LLM API streaming #20244
• Enable SwiGLU clamping and update NVFP4 FC1 weight layout in Rubin CuTe DSL MoE #19488, #20181
• Overhaul the NVLink one-sided all-to-all #19610
• Fix
• Align MiniMax-M3 and GLM reasoning and tool parsing between streaming and non-streaming #20442, #19935
• Flush the Hugging Face streaming detokenizer on the final response #20424
• Restore trtllm-serve streaming throughput by constraining uvloop below 0.23.0 #19992
• Constrain FastAPI to versions below 0.140.0 #20161
• Keep the overlap scheduler responsive while waiting for disaggregated KV receives #19857, #19940
• Reserve retiring-request slot headroom for overlap scheduling without attention data parallelism #20250
• Correct draft KV scheduling and prompt-tail recomputation on cache hits with overlap speculation #19745, #19748
• Size draft-probability buffers by batch capacity and fix speculative beam-search races #19490, #20064
• Size Blackwell speculative-decoding tree masks for the attention kernel layout #20223
• Guard Whisper cross-KV reuse when encoder tokens are unavailable #19302
• Preserve LoRA cache contents when too few pages can be evicted #20212
• Reclaim idle eager CUTLASS MoE workspace #19981
• Correct CuTe DSL MoE autotuning inputs and restore FC2 N-tile autotuning #17410, #20277
• Fix W4A16 NVFP4 MoE dequantization and expert-load-balancing migration, including Nemotron-H #19929
• Restore routing-weight dtype after NCCL expert-parallel dispatch #20081
• Enable SM107 NVFP4 CuTe DSL dispatch and fix CUDA-core GEMM for wide FP8 outputs #20204, #20271
• Enable 96-head MLA in the FMHA autotuner and fix CuTe MLA page-table bounds #18599, #20256
• Prevent the SM120 DSA KV append crash from a missing stride factor #20062
• Bound packed indexer prefill logits and use 64-bit offsets for indexer K-cache #20164, #20190
• Keep DeepSeek-V4 tensor-parallel decode all-reduces off NCCL #20233
• Avoid deep-copying pending media loads in the DeepSeek-V4 tokenizer #20191
• Fence Helix all-to-all shared writes before asynchronous bulk copies #20221
• Match KDA decode fallback layout to prefill #19987
• Preserve indexed GDN state during Triton autotuning and align recurrent Q/K normalization #19859, #19860
• Propagate client import paths to remote MPI workers and refresh stale worker environments #19494, #20151
• Drive MPI progress from a native thread on pipeline-parallel ranks #20030
• Apply phase-appropriate MoE communication timeouts during warmup #20246
• Keep single-rank device-mesh topology separate from the job-wide mesh #20021
• Stop the statistics RPC server when its executor worker shuts down #20158
• Route FLUX.2 linear_in quantization scales correctly when splitting gate and up weights #20403
• Fix correctness issues on Rubin and in Kimi K3, KDA, MNNVL, and DFlash paths #19959, #19988
• Allocate DLPack managed tensors in one block to fix lifetime and shutdown issues #18310
• Reject paged-context FMHA configurations without the required fused context kernel #19746
• Honor models that require KV cache manager V2 instead of falling back to V1 #20093
• Raise a clear error for prompt files with no usable prompts #17733

• Documentation
• Add Qwen3.5 GB300 agentic-serving guidance and curated configurations #20514
• Add the MoE Performance Guide and performance dashboard #19546
• Add a...