Trinity Large Tech Report

Blog | Arcee AI | Trinity Large 400B Open MoE Explained | Arcee AI | Building Open Intelligence

Trinity Large Tech Report: 400B Open MoE Architecture Explained

Trinity Large Tech Report: 400B Open MoE Architecture Explained

+ 3 Lucas Atkins, Varun Singh, Charles Goddard, Mark McQuade

2 min read • Feb 20, 2026

Models Arcee AI Trinity Large Tech Report

Arcee AI has published the technical report for Trinity Large, the largest model in its Trinity family of open-weight Mixture-of-Experts (MoE) language models. Trinity Large has 400B total parameters, with only 13B active per token. The report also covers two smaller models: Trinity Nano (6B total, 1B active) and Trinity Mini (26B total, 3B active). All checkpoints are available on Hugging Face . Why it matters Enterprises want models they can audit, host, and adapt inside their own environments, with clear data provenance and licensing. Agentic and reasoning workloads also produce long outputs over long inputs, so inference efficiency is a hard requirement. Trinity was designed for both. Architecture built for efficiency Trinity Large is extremely sparse: 256 routed experts plus one shared expert, with four routed experts active per token. Attention follows a 3:1 pattern of local sliding-window layers to global layers, which cuts cost at long sequence lengths. Gated attention, QK-normalization, depth-scaled sandwich norm, and sigmoid routing complete the design. The models were trained with the Muon optimizer, which supports larger batch sizes and better sample efficiency than AdamW. Data and training at scale DatologyAI curated the pretraining data, including more than 8 trillion tokens of synthetic web, code, STEM, and multilingual data, making this one of the largest documented synthetic data efforts for pretraining. Trinity Large trained on 17 trillion tokens across three phases that shifted progressively toward code, math, and science. Training ran on 2,048 NVIDIA B300 GPUs managed by Prime Intellect . All three models finished pretraining with zero loss spikes. Two new stability techniques Early Trinity Large runs hit router instability and expert collapse. The team introduced Soft-clamped Momentum Expert Bias Updates (SMEBU), a load-balancing method that replaces fixed-size bias steps with smooth, momentum-dampened updates that settle instead of oscillating. They also built the Random Sequential Document Buffer (RSDB), a data loader that reduces domain imbalance within each batch. In the final training phase, RSDB cut batch heterogeneity by 4.23x and step-to-step loss variance by 2.4x. Long context that extrapolates Trinity Large was extended to 256K tokens by adjusting only the global attention layers. It scored 0.994 on multi-key needle-in-a-haystack at 256K and 0.976 at 512K, a length it never trained on. At 1M tokens it scored 0.42, which signals headroom for future versions. Results Trinity Large Base is competitive with GLM-4.5 Base, despite 4x higher sparsity and roughly 2.5x fewer active parameters. It posted 88.62 on MBPP+, 82.58 on MMLU, and 90.11 on HellaSwag. The instruct model, Trinity Large Preview, scored 87.21 on MMLU, 75.25 on MMLU-Pro, and 63.32 on GPQA Diamond. In FP8 inference tests on 8xH200, Trinity Large delivered higher throughput, faster time to first token, and lower end-to-end latency than DeepSeek-V3 and GLM-4.7 across every tested input/output length. What’s next Post-training was intentionally light, so Trinity Large Preview is a preliminary release. The Arcee research team plans more extensive post-training and flags greater sparsity and larger-batch training as keys to efficient scaling.

Related articles

Models Models Introducing NAC, an Open-Source Harness for Long-Running Agent Work T + 3 4 authors

15 min read • Aug 13, 2026

News News Trinity Large 8 min read • Jan 27, 2026

Models Models Introducing the Arcee Open Models API Beta 1 min read • Aug 13, 2026

Everything you need to start building. Ship on our API or download the models and run them yourself. Every model is open-weight, so you can inspect, fine-tune, and go to production on your terms. Get API Key Talk To Us