22580 Gpt 2 To Kimi K3 Explained

22,580: GPT-2 to Kimi K3, explained Kimi K3 is here. Try it now

Foundations

22,580: GPT-2 to Kimi K3, explained

Twenty-two thousand five hundred and eighty. That’s how many GPT-2 (2019) models fit inside KimiK3 (2026).

Authors

Ali Taha

Last updated July 30, 2026

Share

TL;DR Twenty-two thousand five hundred and eighty. That’s how many GPT-2 (2019) models fit inside Kimi K3 (2026). We scaled up by a factor of 22,580 in seven years. But is it just... scale?

In this worklog, I’ll walk through how we got here and how much, or how little, has actually changed since then. We’ll trace the major architectural developments leading to Kimi K3.

✕

GPT-2 GPT-2 is a decoder-only architecture: tok_emb = self.transformer.wte(idx) # token embeddings of shape (b, t, n_embd) pos_emb = self.transformer.wpe(pos) # position embeddings of shape (t, n_embd) x = self.transformer.drop(tok_emb + pos_emb) for block in self.transformer.h: x = block(x) x = self.transformer.ln_f(x) logits = self.lm_head(x) return logits The input receives token and positional embeddings: ✕

Each transformer block, zoomed in, looks like this: 1 class Block ( nn.Module ): 2 def __init__ ( self, config ): 3 super ().__init__() 4 self.ln_1 = LayerNorm(config.n_embd, bias=config.bias) 5 self.attn = CausalSelfAttention(config) 6 self.ln_2 = LayerNorm(config.n_embd, bias=config.bias) 7 self.mlp = MLP(config) 8 9 def forward ( self, x ): 10 x = x + self.attn(self.ln_1(x)) 11 x = x + self.mlp(self.ln_2(x)) 12 return x ✕

The attention process looks like this: 1 class CausalSelfAttention ( nn.Module ): 2 3 def __init__ ( self, config ): 4 super ().__init__() 5 assert config.n_embd % config.n_head == 0 6 # key, query, value projections for all heads, but in a batch 7 self.c_attn = nn.Linear(config.n_embd, 3 * config.n_embd, bias=config.bias) 8 # output projection 9 self.c_proj = nn.Linear(config.n_embd, config.n_embd, bias=config.bias) 10 # regularization 11 self.attn_dropout = nn.Dropout(config.dropout) 12 self.resid_dropout = nn.Dropout(config.dropout) 13 self.n_head = config.n_head 14 self.n_embd = config.n_embd 15 self.dropout = config.dropout 16 17 def forward ( self, x ): 18 B, T, C = x.size() # batch size, sequence length, embedding dimensionality (n_embd) 19 20 # calculate query, key, values for all heads in batch and move head forward to be the batch dim 21 q, k, v = self.c_attn(x).split(self.n_embd, dim= 2 ) 22 k = k.view(B, T, self.n_head, C // self.n_head).transpose( 1 , 2 ) # (B, nh, T, hs) 23 q = q.view(B, T, self.n_head, C // self.n_head).transpose( 1 , 2 ) # (B, nh, T, hs) 24 v = v.view(B, T, self.n_head, C // self.n_head).transpose( 1 , 2 ) # (B, nh, T, hs) 25 26 # manual implementation of attention 27 att = (q @ k.transpose(- 2 , - 1 )) * ( 1.0 / math.sqrt(k.size(- 1 ))) 28 att = att.masked_fill(self.bias[:,:,:T,:T] == 0 , float ( '-inf' )) 29 att = F.softmax(att, dim=- 1 ) 30 att = self.attn_dropout(att) 31 y = att @ v # (B, nh, T, T) x (B, nh, T, hs) -> (B, nh, T, hs) 32 y = y.transpose( 1 , 2 ).contiguous().view(B, T, C) # re-assemble all head outputs side by side 33 34 # output projection 35 y = self.resid_dropout(self.c_proj(y)) 36 return y Once the final hidden-state matrix is produced, the language-model head maps it into vocabulary logits. During autoregressive decoding, only the logits at the final position are needed to select the next token. This is an inefficiency of decoder-only generation: the model computes representations for every input position, but each decode step consumes only the final position’s logits. Without caching, much of that work would be repeated for the next token. ✕

Without a cache, we then repeat the process for token n+1. KV Cache Invention The KV cache comes from a straightforward observation: after appending the generated token to the input, the model would otherwise recompute projections for all previous tokens. Storing their key and value vectors avoids that redundant work. That storage is the KV cache. It retains vectors for the previous N-1 tokens and can become large enough to create a memory-bandwidth bottleneck. ✕

Size and scale Overall, with about 50k possible tokens, 12 blocks, 12 heads, and an embedding dimension of 768, our baseline model is about 124M parameters. vocab_size: int = 50304 # GPT-2 vocab_size of 50257, padded up to nearest multiple of 64 for efficiency n_layer: int = 12 n_head: int = 12 n_embd: int = 768 At 2.8 trillion parameters, one Kimi K3 model contains roughly as many parameters as 22,580 GPT-2 models. Linear attention Softmax attention applies its nonlinearity after the q·k product, coupling every query to every key. Linear attention instead applies a feature map, such as ELU+1, to q and k separately. This makes the product reassociable, so the growing set of K and V vectors can be folded into a fixed D×D state. The paper’s O(N²) framing is easy to misread without its 2020 context. At the time, training commonly materialized the full N×N attention matrix, FlashAttention did not exist, and reference autoregressive implementations often recomputed the token history without a KV cache. ✕

A KV cache avoids recomputation, but each decode step still reads all previous keys and values. The per-step cost grows as O(ND), the cache grows as O(ND), and generating N tokens costs O(N²D) in total. In practice, repeatedly streaming that cache from HBM is the bottleneck. Linear attention replaces the growing cache with a fixed D×D state. Across N tokens, the work becomes O(ND²). Decode remains sequential, but when N is much larger than D, the fixed state can reduce memory traffic and deliver substantial speedups. Claims of thousand-fold improvements generally compare against the older cacheless baseline; gains over a modern KV-cached implementation are more modest. The implementation difference is easiest to see by starting with the conventional KV cache: 1 def forward ( self, x, mask= None , past_kv= None ): 2 # x is b,t,d 3 b,t,d=x.shape 4 d_head=d//self.num_heads 5 h=self.num_heads 6 qkv=self.qkv_proj(x) 7 8 q=qkv[:, :, :d].view(b,t,h,d_head).transpose( 1 , 2 ) 9 k=qkv[:, :, d: 2 *d].view(b,t,h,d_head).transpose( 1 , 2 ) 10 v=qkv[:, :, 2 *d:].view(b,t,h,d_head).transpose( 1 , 2 ) 11 12 # at prefill, q,k,v have shapes b,h,t,d 13 # at decode, shape is b, h, 1, d 14 # so i cat at the t dimension, dim(2) 15 16 if past_kv is not None : 17 k_past=past_kv[ 0 ] 18 v_past=past_kv[ 1 ] 19 k=torch.cat((k_past, k), dim= 2 ) 20 v=torch.cat((v_past,...