Tencent-Hunyuan/ExplorationBench

Tencent-Hunyuan/ExplorationBench

Language: Python

License: NOASSERTION

Stars: 2

Forks: 0

Open issues: 0

Created: 2026-09-30T10:51:12Z

Pushed: 2026-10-08T06:59:59Z

Default branch: main

Fork: no

Archived: no

README:

中文 ｜ English

ExplorationBench: Measuring AI Systems' Exploration in Verifiable Alien Worlds

🖥️ Official Website | 📮 Evaluate Your Model

---

🔥 News

• 2026.09: 🎉 ExplorationBench released — two verifiable Alien Worlds with 55 discovery targets and 140 held-out tasks, and results for 10 frontier AI systems. [[Paper]](https://arxiv.org/abs/2609.30199) [[Website]](https://explorationbench.com) [[Blog]](https://explorationbench.com/blog/)

📋 Contents

• [Overview](#-overview)
• [Leaderboard](#-leaderboard)
• [Key Findings](#-key-findings)
• [What Is Released](#-what-is-released)
• [Quick Start](#-quick-start)
• [Repository Structure](#-repository-structure)
• [Evaluate Your Model](#-evaluate-your-model)
• [Related Benchmarks](#-related-benchmarks)
• [License](#-license)
• [Citation](#-citation)
• [Contact](#-contact)

📄 Overview

Scientific discovery begins where known problems end. There, AI systems must explore: frame hypotheses, design experiments, and iterate on the results. ExplorationBench measures this ability in verifiable Alien Worlds. Their rules are executable, so every answer is checked exactly, and they conflict with familiar knowledge, so recall alone cannot solve the tasks.

A system starts from a flawed manual and a fixed set of worked examples. Over four rounds it probes the world through a single tool, and the world answers with deterministic feedback and no explanation. After every round, a copy of the conversation with tools disabled reports the rules it believes hold and answers every held-out task three times. The copy is then discarded, so test items never leak into exploration.

*The evaluation protocol. Every system starts from the same flawed manual and the same fixed worked examples and chooses its own probes over four rounds. At every milestone Mt, a tool-disabled copy reports the rules it believes hold and answers the held-out tasks.*

*Held-out accuracy over autonomous exploration. Top: AlienCode; bottom: AlienLogic. Each cell highlights one system's M0–M4 curve; the other nine are grey. Each curve is the system's Best@3 trajectory, and each milestone is the mean of three answers per held-out task.*

Stats: 2 sandboxes · 55 discovery targets · 140 held-out tasks · 10 frontier systems · 3 independent trajectories per system and sandbox · every verdict from an interpreter or a proof checker, with no LLM judge

Sandboxes:

• AlienCode — program synthesis in a small calculation language whose familiar-looking operators follow hidden semantics: SHATTER multiplies, EMIT(100) prints 127, and no primitive adds. 31 discovery targets and 70 held-out tasks; each program is graded by an interpreter on five private inputs.
• AlienLogic — formal proof in a first-order natural-deduction system with 24 patched inference rules. 70 held-out theorems, 23 of them unprovable; a proof checker verifies every proof, and an unprovable theorem scores only when the system declines to prove it.

🏆 Leaderboard

Held-out accuracy (%) after four rounds of autonomous exploration (M4). Best@3 is the best of three independent trajectories and Mean@3 their mean; M0 is the Best@3 trajectory's accuracy after the worked examples alone. Every score is the mean of three answers per task, and every system runs at its highest reasoning setting. More columns, the control conditions, and interactive trajectories are on the website.

AlienCode

| Rank | System | M0 | Best@3 | Mean@3 | |:---:|---|---:|---:|---:| | 1 | Claude Opus 5 | 3.8 | 89.0 | 73.5 | | 2 | GPT-5.6 Sol | 11.0 | 88.1 | 79.5 | | 3 | Gemini 3.8 Flash | 1.4 | 79.0 | 39.7 | | 4 | Kimi K3 | 2.4 | 79.0 | 38.7 | | 5 | Grok 4.6 | 4.8 | 69.0 | 53.2 | | 6 | Qwen3.8-Max | 2.4 | 65.7 | 64.0 | | 7 | Hy4 preview | 0.5 | 63.3 | 30.2 | | 8 | DeepSeek-V4.1-Flash | 1.4 | 59.5 | 46.3 | | 9 | Seed2.1 Pro | 2.4 | 40.0 | 24.8 | | 10 | DeepSeek-V4-Pro | 1.9 | 12.9 | 7.3 |

AlienLogic

| Rank | System | M0 | Best@3 | Mean@3 | |:---:|---|---:|---:|---:| | 1 | Grok 4.6 | 42.9 | 81.0 | 72.1 | | 2 | Claude Opus 5 | 43.3 | 79.5 | 75.9 | | 3 | Qwen3.8-Max | 45.2 | 77.6 | 74.3 | | 4 | GPT-5.6 Sol | 51.0 | 76.7 | 75.1 | | 5 | Hy4 preview | 35.2 | 76.2 | 69.2 | | 6 | DeepSeek-V4-Pro | 33.8 | 75.2 | 66.3 | | 7 | Kimi K3 | 45.2 | 74.3 | 69.4 | | 8 | DeepSeek-V4.1-Flash | 51.4 | 73.8 | 60.6 | | 9 | Seed2.1 Pro | 37.6 | 69.0 | 59.8 | | 10 | Gemini 3.8 Flash | 45.7 | 59.5 | 59.0 |

💡 Key Findings

• Exploration, not recall or thinking alone, produces the knowledge the tasks require. No AlienCode trajectory exceeds 15.7% after the worked examples; after four rounds the best reaches 89.0%. The same number of rounds without environment feedback stays at 0.5–11.0%.
• Exploration works best when the system designs its own experiments. Replaying a system's own best probes to it gives it exactly the same evidence, yet in AlienCode autonomous exploration still does better for 9 of 10 systems, by a median of 17.4 points.
• The two worlds get stuck in different places. In AlienLogic, being told the rules yields 96–100%, above every system's best trajectory, so discovery is the bottleneck. In AlienCode, the best trajectory beats being told the rules up front for 7 of 10 systems, so using the rules is.
• Knowing a rule does not guarantee using it correctly. Tasks whose required rules the final rule report states correctly are still solved only 73.4% of the time.
• A single score hides a lot. One system's trajectories under the same budget can end tens of points apart (Kimi K3: 5.7–79.0% in AlienCode), a system's rank in one world barely predicts its rank in the other (Spearman 0.35), and gains arrive in leaps that later rounds can undo.

📦 What Is Released

A world whose rules are public measures recall instead of exploration, so the rules and test instances of the two evaluation worlds are kept private. Everything else needed to understand and run the benchmark is here:

• The evaluation code, as it ran for the paper: harnesses, protocol, scoring, control conditions, three answers per task, open-book answering, rescoring, and audits. Only the world-specific modules differ, and...