ByteDance-Seed/VeOmni v0.1.12

v0.1.12

Repository: ByteDance-Seed/VeOmni

Tag: v0.1.12

Published: 2026-09-09T03:57:51Z

Prerelease: no

Release notes:

Highlights

• DeepSeek-V4, end to end — training support (#840), TileLang kernels (#912), Liger kernel path (#946), MFU accounting (#944), Ulysses sequence parallel (#949) and context parallel for attention, compressors and indexer (#1111, #1131), the Lightning Indexer KL objective (#1134) with its TileLang teacher-distribution kernel (#1109), FlashMLA cuDNN DSA backend (#850), fp8/fp4 quantized weight export (#962), QAT fake-quant training (#1089), and NPU support (#974).
• New models — MiniMax H3 (#1075) with Ulysses SP (#1120), LTX-2.3 (#858, #859), GPT-OSS (#854) plus a 120B EP4 LoRA recipe (#921), and an expert-parallel plan for Qwen3.5-MoE causal LM (#1106).
• LoRA — native PEFT-free LoRA stack with MoE-LoRA and expert parallelism (#758), VLM trainer support (#1018), Qwen3.5-MoE (#1094), EP-sharded streaming load for PEFT models (#915), and faster eager MoE-LoRA routing (#1154, #1155).
• Parallelism and performance — per-module local parallel state via context manager (#893), FSDP2 torch.compile (#881, #1009), MindSpeed-style async activation offload (#965), shared async Ulysses backward helpers (#1113), FlexAttention with Ulysses (#945), MagiAttention on SM90 with Ulysses (#1021, now an optional extra in #1160), HSDP for expert parallel (#867), optional train-step sync (#938), and reused loss denominators (#941).
• Muon — Gram Newton-Schulz backends (#953), FSDP2 all-to-all optimization (#973), head-group split (#980) with qualified module names (#1108), and Qwen3.5-MoE Muon on Ascend (#1092).
• Ascend / NPU — fused npu_rms_norm (#978), AscendC chunk gated delta rule (#924), Triton clamped SwiGLU (#1104), Qwen3-Omni-MoE with fused RoPE (#910), CANN 9.1.0 images (#1128), and GDN Dockerfiles (#1084).
• More accelerators — AMD ROCm (#890) and Cambricon MLU (#903).
• Data and trainer — effective-token dynamic batching (#833), map-style datasets in worker-side dynamic batching (#853), dataloader scheduling knobs (#940), model aux metrics reported without folding them into the loss (#1110), channel loss logging (#892), checkpoint early-stop (#926), and a pending async save awaited at train end (#1162).
• Toolchain — transformers 5.9.0 is the default install (#774), and CUDA 13.0 + PyTorch 2.11 images (#827).

Breaking Changes

• Per-module local parallel state moves to a context manager (#893)
• DeepSeek-V4 kernel naming unified (#934)
• SeedOmni V1 model stack removed (#1082)
• ChunkMBS removed (#1119)
• Head-split Muon is selected by qualified module names instead of a DSA indexer flag (#1108)

What's Changed

• [ckpt, trainer] fix: avoid HBM OOM during DCP save under MoE / VLM training by @TimYangst in https://github.com/ByteDance-Seed/VeOmni/pull/791
• [model] feat: register ForTokenClassification in model registries by @Luosuu in https://github.com/ByteDance-Seed/VeOmni/pull/795
• [parallel] fix stride error in npu when dsp+recompute by @yicheng-gong in https://github.com/ByteDance-Seed/VeOmni/pull/793
• [ckpt, data] refactor: remove _fill_missing_optimizer_states, lazy-import multimodal deps by @Luosuu in https://github.com/ByteDance-Seed/VeOmni/pull/800
• [ckpt] refactor: deduplicate ExtraParallel dim preprocessing by @Luosuu in https://github.com/ByteDance-Seed/VeOmni/pull/803
• [ckpt] refactor: centralize async DCP save lifecycle in wait_for_pending_save() by @Luosuu in https://github.com/ByteDance-Seed/VeOmni/pull/802
• [parallel] fix: keep root's auto-no-reshard so fused-linear kernels can backward by @Luosuu in https://github.com/ByteDance-Seed/VeOmni/pull/806
• [model, docs] refactor: make patchgen reusable as a library by @TimYangst in https://github.com/ByteDance-Seed/VeOmni/pull/807
• [ci, docker] chore: bump uv from 0.9.8 to 0.11.16 by @TimYangst in https://github.com/ByteDance-Seed/VeOmni/pull/782
• [model, ci, docs] feat: bump transformers to 5.9.0 by @TimYangst in https://github.com/ByteDance-Seed/VeOmni/pull/774
• [config, ci, docs] chore: relax uv pin from ==0.11.16 to a 0.9.8-0.11 range by @TimYangst in https://github.com/ByteDance-Seed/VeOmni/pull/808
• [model, ci, docs, agent] refactor: ship patchgen as a sibling standalone package by @TimYangst in https://github.com/ByteDance-Seed/VeOmni/pull/809
• [data, model] feat: Implemented dynamic batch logic optimization by @zhihaofang1017 in https://github.com/ByteDance-Seed/VeOmni/pull/789
• [ckpt, model] fix: rename per-expert HF weight_map keys for fused MoE HF export by @Coach257 in https://github.com/ByteDance-Seed/VeOmni/pull/799
• [docs, docker] feat: Update A3 Docker image to 9.0.0 and NPU model support table by @cls1206 in https://github.com/ByteDance-Seed/VeOmni/pull/817
• [model] fix: derive hybrid-attn layer counts from layer_types for transformers v5 by @TimYangst in https://github.com/ByteDance-Seed/VeOmni/pull/819
• [docker] feat: update docker with template by @FoolPlayer in https://github.com/ByteDance-Seed/VeOmni/pull/814
• [docs] chore: drop stale offline MoE merge step from Qwen3 / Qwen3-MoE / Qwen3-Omni recipes by @TimYangst in https://github.com/ByteDance-Seed/VeOmni/pull/813
• [data, omni, docs] feat: Qwen3-Omni recipe for offline-extracted audio-enabled video by @TimYangst in https://github.com/ByteDance-Seed/VeOmni/pull/812
• [ops, ci] feat: extend NPU kernel test coverage and add CPU-runnable registry/eager sanity tests by @Kirrito-k423 in https://github.com/ByteDance-Seed/VeOmni/pull/820
• [data] feat: align StatefulDataLoader snapshot frequency to save_steps by @Fazziekey in https://github.com/ByteDance-Seed/VeOmni/pull/822
• [docs] fix: Ascend documents fix by @dzy00897185 in https://github.com/ByteDance-Seed/VeOmni/pull/830
• [model] feat: support sp on Qwen-Image by @FoolPlayer in https://github.com/ByteDance-Seed/VeOmni/pull/815
• [perf] fix: enable ProfilerWithMem with NPU by @yanghw116 in https://github.com/ByteDance-Seed/VeOmni/pull/831
• [docker] feat: support cu130+pt211 by @FoolPlayer in https://github.com/ByteDance-Seed/VeOmni/pull/827
• [model] fix: patched forward docstrings for transformers 5.9 by @Luosuu in https://github.com/ByteDance-Seed/VeOmni/pull/836
• [misc] feat: pin FA4 4.0.0b16 by @Luosuu in https://github.com/ByteDance-Seed/VeOmni/pull/837
• [model] fix: document qwen3 moe cache position by @Luosuu in https://github.com/ByteDance-Seed/VeOmni/pull/838
• [ci] fix: npu aarch64 dependency...