Kimi-K2-Thinking — the open agentic-research bet

Primary source ↗ · Synthesized by the onlylabs Content Studio agent (Claude Code) · web-verified

Kimi-K2-Thinking — the open agentic-research bet

An eval-founder read of moonshotai/Kimi-K2-Thinking (Moonshot AI, modified-MIT, released 2025-11-04; tech blog). Numbers are the card's own self-reported figures — the §1 table is the cross-vendor comparison set; a few heavy / w/ python variants cited in the text are additional settings the same card reports. onlylabs signal.

0. What this is

• An open "thinking agent" — 1T-param MoE, native INT4, 256k context. Kimi K2 Thinking is end-to-end trained to interleave chain-of-thought with tool calls, and its headline claim is stable long-horizon agency: 200–300 sequential tool invocations without drift (where, it says, prior models "degrade after 30–50 steps").
• Native INT4 via quantization-aware training — a claimed lossless 2× low-latency speed-up, baked into post-training rather than bolted on. The differentiator is as much efficiency + horizon as raw score.
• It publishes a full cross-vendor comparison vs GPT-5 (High), Claude Sonnet 4.5 (Thinking), DeepSeek-V3.2, Grok-4, and its own prior K2 0905 (which shows the gen-over-gen jump).

1. The benchmark profile (self-reported, by setting)

The setting column matters as much as the score — tool use swings results enormously.

2. Where it leads, where it trails

• Leads — agentic web research, decisively. BrowseComp 60.2 (GPT-5 54.9, Sonnet 24.1, DeepSeek 40.1), Seal-0 56.3 and Frames 87.0 are all the table's best. With tools, HLE 44.9 (heavy mode 51.0) tops GPT-5. This is the "thinking agent that browses for hundreds of steps" thesis landing — and the gap over Sonnet on BrowseComp (60.2 vs 24.1) is the single most striking number on the card.
• Competitive — math + agentic SWE. AIME25 94.5 ≈ GPT-5 94.6; GPQA 84.5 within a few points of the field (GPT-5 85.7, Sonnet 83.4); Terminal-Bench 47.1 between GPT-5 (43.8) and Sonnet (51.0).
• *Trails — raw coding, knowledge, and no-tools reasoning. SWE-bench Verified 71.3 (Sonnet 77.2, GPT-5 74.9), LiveCodeBench 83.1 (GPT-5 87.0), MMLU-Pro 84.6 (Sonnet 87.5), HealthBench 58.0 (GPT-5 67.2), and HLE no-tools 23.9 (GPT-5 26.3). Strip the tools and the edge narrows — the model's value is in the agentic loop*, not the base weights.

3. Comparability gotchas (the setting is the benchmark)

• Tool setting dominates. HLE goes 23.9 → 44.9 → 51.0 across no-tools / w-tools / heavy — a 2× swing on the same benchmark. AIME25 goes 94.5 → 99.1 (w/ python) → 100.0 (heavy). A bare "HLE score" is meaningless without the setting; the card is unusually honest in showing all three.
• * = sourced/measured differently (the card marks many competitor cells) — so the columns aren't one uniform harness.
• The real differentiators aren't in the table. "200–300 tool calls without drift" and "native INT4, 2× speed-up" are durability and efficiency claims, not single-number benchmarks — and they're the actual reasons to care, more than a 3-point SWE delta.
• K2 0905 is the prior K2, included to show the jump (e.g. BrowseComp 7.4 → 60.2) — useful, but it inflates "beats N models" headline reads if you don't notice one "competitor" is its own predecessor.

4. What it means for an eval / RL-environments founder

1. *Open agentic research is a distinct frontier from agentic engineering. Where GLM-5.2 bets on long-horizon coding, K2 Thinking bets on long-horizon browsing/search (BrowseComp, Seal-0, Frames) — and an open model now holds the top of that category. Two different open-weight agentic frontiers, two different eval stacks. 2. The benchmark that matters is a horizon benchmark. "200–300 sequential tool calls without drift" is the claim the field can't yet measure with a single number — the gap between a 50-step and a 300-step agent is exactly where current evals are thinnest. That's the eval product to build. 3. Efficiency is now a first-class eval axis.* Native-INT4-at-no-quality-loss reframes "score per token" into "score per dollar/latency." A vendor-neutral harness that reports cost/latency alongside accuracy would capture what these cards increasingly compete on.