Trinity Large

Arcee AI | Trinity Large: An Open 400B Sparse MoE Model

Trinity Large Thinking: Available on OpenRouter.

Try now ↗

ENTERPRISE

Research

COMPANY

Get API

Blog / Trinity Large

Trinity Large Lucas Atkins ,

•

January 27, 2026

A deep dive into Trinity Large, covering architecture, sparsity, training at scale, and why we shipped Preview, Base, and TrueBase checkpoints.

Two months ago I wrote about why we decided to stop treating pretraining like someone else's job. At the time, Trinity Nano Preview and Trinity Mini had just released, and Trinity Large had started training. We were in the middle of our first run so big that you either laughed or got nauseous. Frankly, I felt either we’d end up with a really great base model or fall flat on our faces with a tired wallet. Little did I know, we’d get both. Here’s what we’re shipping, what surprised us, what broke, and what it took to make a 400B sparse MoE behave. We're putting out three variants: Trinity-Large- Preview is lightly post-trained and chat-ready, Trinity-Large- Base is our best pretraining checkpoint after the full 17T recipe, and TrueBase is an early checkpoint from the same run at 10T tokens, without any instruct data or LR anneals. What many would consider a true base model. Trinity-Large is a 400B parameter sparse MoE with 13B active parameters per token . It uses 256 experts with 4 experts active per token . That sparsity ratio is pretty high compared to our peers, save for Llama-4-Maverick:

Model Routing (k-of-N) Routing fraction

Trinity Large 4-of-256 1.56%

DeepSeek-V3 8-of-256 3.13%

MiniMax-M2 8-of-256 3.13%

GLM-4.5 8-of-160 5.0%

Qwen3-235B-A22B 8-of-128 6.25%

Llama 4 Maverick 1-of-128 0.78%

We originally aimed for a slightly different total size (420B), but we ended up increasing the number of dense layers (from 3 to 6) to help keep routing stable at this sparsity.

Trinity-Large-Base is a true frontier-class foundation model. We match and exceed our peers in open-base models across a wide range of benchmarks, including math, coding, scientific reasoning, and raw knowledge absorption. Inference efficiency We trained on 2048 Nvidia B300 GPUs. As far as we can tell, it’s the largest (publicly stated, at least) pretraining run done on these machines. That means two things: They’re wicked fast. They’re not cheap.

Therefore, we had to make the most of the money we allotted to these machines, which was just over 30 days. Ridiculously fast for a run of this scale, so efficient training was the name of the game. Hence, the level of sparsity referred to above. Combined with our efficient attention outlined in our technical report , this enabled us to train and, by extension, run inference much faster than our peers in the same weight class. All while not sacrificing performance. Roughly 2-3x faster for the same hardware.

Momentum-based expert load balancing We keep MoE routing under control by nudging each expert’s router bias up or down depending on whether that expert is being over- or under-used. The update is capped with a tanh clip so it stays bounded, and we add momentum to smooth it across steps and avoid step-to-step ping-pong. On top of that, we include a small per-sequence balance loss so load is not only balanced in expectation across the batch, but also within individual sequences. z-loss We use z-loss to stop the LM-head logits from drifting upward during training. It is a lightweight regularizer that keeps logit scale from creeping up without bound. We also log basic logit stats (for example max and mean) as a simple early warning for instability. The exact equations are in the technical report . Our fastest configuration was HSDP with expert parallelism set to 8, which gave us 2048 data-parallel ranks. In that setup, we pushed throughput further by increasing batch size after 5T tokens of training. We were comfortable doing that because the model is highly sparse, Muon supports a larger critical batch size than AdamW, and the MiniMax-01 paper suggests batch-size scaling remains workable in this regime. In the main run, once we had stability dialed in, the loss curve stayed smooth the whole way through. You can see clear phase transitions, no spikes, and a steady march to the end.

The full pretraining run finished in 33 days. That's pre-training only; it doesn't include context extension or post-training. Data Trinity Large was trained on 17T tokens of data curated by DatologyAI , split across three phases of 10T, 4T, and 3T tokens. This mix uses state-of-the art programming, STEM, reasoning, and multilingual data curation, targeting 14 non-English languages. Notably, over 8 trillion tokens of synthetic data were generated for this dataset across web, code, math, reasoning, and multilingual domains, using a breadth of state-of-the-art rephrasing approaches. A less advanced version of this curation approach worked well for smaller models like Trinity Nano and Trinity Mini, but we wanted to shoot for the moon with Trinity Large. So DatologyAI delivered a number of curation advancements specifically for inclusion into Trinity Large. There’s always a leap of faith when doing something for the first time in public, but the effectiveness of the curation approach as a whole is reflected in the downstream evaluations, where the Trinity Large-Base demonstrates frontier-level performance across the targeted capability domains. Trinity-Large-Preview The preview we’re releasing today is not a reasoning model. A benefit beyond production inference efficiency is that it also carries over into RL, enabling quicker rollouts for a given total parameter size. But it’s equally as sensitive as pretraining was, and while it’s currently undergoing further post-training and will be a full-fledged reasoning model upon release, we believe there’s a fine line between intelligence and usefulness , and while the reasoning variant is very intelligent, it needs longer in training before it becomes maximally useful given the extra tokens per output. As such, to maximize usefulness and provide an early checkpoint, Preview is a non-reasoning, or “instruct,” model. It’s a particularly light post-training, as most of our compute went towards pre-training and is continuing with the reasoning version. It is an extremely capable model for its size and gave us the opportunity to flex some old-school post-training muscles we haven’t had the chance to flex in quite some time. It excels in creative writing,...