Glm 53

GLM 5.3: Scaling with post-training, intuitively explained Try GLM-5.3 today. Frontier intelligence at a fraction of the cost. Here

AI models

GLM 5.3: Scaling with post-training, intuitively explained

GLM 5.3's gains came entirely from post-training. An intuitive look at the environment design, RL algorithms, and infrastructure behind the leap.

Authors

Chloe Florit

Alex Ker

Last updated August 28, 2026

Share

Sutton's bitter lesson said general-purpose methods that scale with compute will win. GLM-5.3 is a perfect application of this law to RL, and it means a model can leapfrog its predecessor without a new pre-training run: an identical base model (GLM-5.2) improved over 50% through scaled post-training compute, deliberate environment design, and efficient infrastructure engineering. In this post, we'll look at how environment design led to GLM-5.3. Then we'll look at how the architecture behind GLM 5.2 carried over to enable efficient training and inference possible. Lastly, we'll cover the RL algorithm and infrastructure, specifically SAO and slime, that turned those cheap rollouts into stable, large-scale post-training. Environment design The focus of GLM 5.3’s post-training work comes down to realistic environment design: creating tasks that mirror expert work rather than leetcode-style problems. For example, ML infrastructure problems mimic those that a human engineer would solve. A model is given access to the same resources as a human engineer, including access to compute clusters, storage, and docs. The task is to identify bottlenecks in the training stack, then optimize them by running experiments and iterating on speedups while maintaining correctness. The environment generation is also agent-first. Research agents convert task patterns from real-world workflows into runnable, long-horizon environments (task setups where an agent must work through many steps). A judge agent then checks that each environment is solvable so no bad training signal gets added. Separately, a verifier is automatically generated for each environment without getting access to the reference solution. ✕ Closing reward shortcuts: the three-check verifier test To prevent reward hacking (common in RL), the team also used solver trajectories to close reward shortcuts and trivial solutions. Each verifier is tested before use. It must pass three checks: given the known-correct solution, it awards reward (the oracle check); given a run where the agent did nothing, it awards none; and given a run where the task was left unfinished, it awards none. A verifier that rewards only correct, complete solutions is what makes the training signal trustworthy. In short, this pipeline generates synthetic environments at a high volume and quality, which are critical to the success of post-training. Breaking down GLM 5.2 & 5.3’s architecture ✕ GLM 5.2: Model Architecture GLM-5.3’s architecture is entirely carried over from GLM 5.2, which uses a Mixture of Experts (MoE) architecture: 744B total parameters, but only ~40B active per token.  Each MoE layer has 256 experts, routing just 8 experts per token, so you get the reasoning capacity of a massive model at a lower compute cost.

Multi-head Latent Attention: shrinking the KV cache What&#x27;s a KV (Key-Value) cache? "Keys" help the model figure out which words to pay attention to, and "values" determine what information gets added to a word&#x27;s meaning based on the context. Together, they&#x27;re cached as the "KV cache," so the model doesn&#x27;t recompute them for every new token. In standard multi-head attention, every head keeps its own unique KV pair for every token. Multi-head Latent Attention (MLA) compresses each token&#x27;s keys and values into a small shared latent vector (one per token, per layer), and only this latent is stored in the KV cache. When computing attention, lightweight projection matrices reconstruct (up-project) the per-head keys and values from the latent. This shrinks the KV cache, keeping memory use low as context grows and enabling longer context windows without sacrificing quality. DeepSeek Sparse Attention: making attention efficient DeepSeek Sparse Attention (DSA) has two main components. First, a tiny lightning indexer scores all past tokens for relevance to the current query token. Then a token selector picks the top‑k previous tokens for the MLA to run on. The indexer and selector are both learned. Together, they create an attention mask that filters out less relevant tokens. One of the biggest bottlenecks for running LLMs is that multihead attention is a quadratic, O(N^2), operation. However, since the lightning indexer is in FP8 with a handful of heads, its nominal quadratic cost is dwarfed by the savings from not doing full MLA on the entire context. End‑to‑end cost per token is almost flat out to 128k during prefilling, and only grows linearly in k during decoding, on the order of O(N*k) where k<<N. This makes long reasoning chains affordable. We explained this architecture originally introduced in the DSV3.2 release last year. Multi-token prediction: high acceptance speculative decoding MTP is a form of speculative decoding. Speculative decoding is a technique used to speed up LLM inference by using a smaller, faster draft model to predict multiple tokens in parallel, which the larger, more accurate LLM then verifies. If the target model generated those same tokens itself, it would have to do so sequentially, one token at a time. GLM 5.2 takes a different approach with MTP. Instead of using a separate draft model, GLM-5.2 adds a single lightweight MTP layer on top of its main model. The MTP layer is one transformer block whose weights are reused for every draft step (one step per draft token). At inference, this layer takes the main model&#x27;s final hidden state (which encodes the full context) plus the embedding of the just-sampled token. But since the MTP layer is a transformer block, its attention still needs to look back over the context (one summary vector isn&#x27;t enough). Rather than computing its own keys and values over the context, it reuses the main model&#x27;s KV cache and sparse-attention indices. GLM 5.3’s MTP allows better draft acceptance over other methods like EAGLE. It sequentially drafts up to 5 tokens ahead, which the main model verifies in one parallel forward pass, accepting ~4.5 tokens per pass on average. This outperforms classic speculative decoding with a...