MoonshotAI/PerceptionBench

MoonshotAI/PerceptionBench

Description: PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models

Language: Python

License: Apache-2.0

Stars: 58

Forks: 6

Open issues: 0

Created: 2026-07-23T07:36:15Z

Pushed: 2026-07-27T19:40:42Z

Default branch: master

Fork: no

Archived: no

README:

PerceptionBench

PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models

HomePage | Paper | 🤗 Dataset | Code

Abstract

We introduce PerceptionBench, a benchmark specifically designed to evaluate the atomic visual perception capabilities of Multimodal Large Language Models (MLLMs). Existing benchmarks often fail to isolate perception: holistic evaluations conflate perceptual errors with failures in reasoning or domain knowledge, while application-driven benchmarks only cover narrow, fragmented domains shaped by heuristic designs. To address these limitations, PerceptionBench adopts a bottom-up approach: by diagnosing the earliest failure points in the response of frontier MLLMs across 42 existing benchmarks, we construct an error taxonomy whose perception branch defines ten atomic perceptual capabilities. Guided by this taxonomy, we construct 3,000 verified questions with short, unambiguous answers, each isolating a single capability, with difficulty stemming from perception rather than reasoning or knowledge. Benchmark results across sixteen frontier MLLMs reveal that atomic perception remains largely unsolved—no model reaches 60% accuracy, perception-related hallucination is the weakest capability on average, and similar overall scores conceal sharply divergent capability profiles. PerceptionBench thus provides a capability-level standard for measuring and diagnosing the visual perception boundaries of MLLMs.

Dataset Statistics

The released benchmark comprises 3,000 verified questions across the ten atomic perceptual capabilities. By construction source, 1,800 (60%) are atomic sub-questions decomposed from attributed failures on the source benchmarks, while the remaining 1,200 (40%) are newly authored on supplemented images. The 3,000 questions are subsampled with capability-level balancing and difficulty stratification from the constructed portion of an in-house pool of 17,000+ verified samples.

Leaderboard

Sixteen frontier MLLMs (ten proprietary, six open-source) are evaluated with unified prompts and the highest available reasoning budget. All questions are open-ended with short, uniquely determined answers; GPT-oss-120B judges each response against the reference (agreement with human judgment: 99.7% on a 300-sample audit). Scores are accuracy (%).

| # | Model | Overall | VRel | Count | Attr | Depth | Loc | Comp | FGR | Ctx | OCR | Hallu | |--:|-------|--------:|-----:|------:|-----:|------:|----:|-----:|----:|----:|----:|------:| | 1 | GPT-5.6-Sol | 59.7 | 69.7 | 62.4 | 62.1 | 55.5 | 76.7 | 67.0 | 55.9 | 60.0 | 54.9 | 26.9 | | 2 | Kimi K3 | 58.5 | 68.2 | 59.7 | 59.4 | 52.4 | 70.3 | 59.1 | 55.9 | 53.3 | 61.2 | 41.7 | | 3 | Claude-Fable-5 | 57.2 | 58.5 | 52.9 | 60.9 | 51.5 | 70.4 | 56.1 | 51.6 | 59.8 | 64.3 | 45.0 | | 4 | Gemini-3.1-Pro | 56.2 | 58.8 | 56.9 | 61.8 | 50.0 | 52.7 | 61.7 | 54.8 | 61.2 | 64.3 | 40.6 | | 5 | GPT-5.5 | 55.8 | 61.9 | 55.8 | 60.9 | 48.8 | 65.8 | 65.6 | 47.2 | 58.0 | 56.5 | 34.7 | | 6 | Seed-2.1-Pro | 55.0 | 57.6 | 51.2 | 58.2 | 43.6 | 50.0 | 59.5 | 56.6 | 60.4 | 66.7 | 49.8 | | 7 | Gemini-3.5-Flash | 52.0 | 53.6 | 43.6 | 54.5 | 49.7 | 50.6 | 54.8 | 51.7 | 53.3 | 59.6 | 50.6 | | 8 | Qwen3.7-Plus | 51.1 | 59.1 | 53.3 | 55.8 | 48.5 | 52.7 | 55.9 | 46.8 | 52.2 | 54.5 | 29.5 | | 9 | Qwen3.5-397B-A17B | 47.5 | 55.2 | 49.1 | 53.0 | 44.6 | 46.7 | 49.8 | 44.8 | 50.2 | 52.9 | 26.9 | | 10 | Claude-Opus-4.8 | 47.2 | 51.4 | 44.2 | 49.4 | 40.6 | 58.8 | 48.4 | 40.7 | 44.7 | 54.1 | 38.7 | | 11 | Kimi K2.6 | 42.6 | 50.9 | 45.2 | 43.6 | 42.4 | 45.2 | 39.1 | 34.5 | 40.4 | 40.8 | 41.0 | | 12 | Grok-4.5 | 41.0 | 47.0 | 35.2 | 39.4 | 41.2 | 39.7 | 43.7 | 36.2 | 39.6 | 43.9 | 44.7 | | 13 | Gemma-4-31B | 40.7 | 42.7 | 33.9 | 40.3 | 39.1 | 44.9 | 43.7 | 39.0 | 45.9 | 46.7 | 32.1 | | 14 | GLM-5V-Turbo | 39.6 | 41.2 | 40.0 | 41.2 | 41.2 | 43.9 | 45.2 | 36.6 | 32.9 | 43.5 | 28.0 | | 15 | Minimax-M3 | 33.1 | 40.0 | 30.3 | 34.6 | 36.7 | 33.3 | 31.2 | 26.6 | 31.0 | 35.7 | 29.9 | | 16 | GLM-4.6V | 32.5 | 35.2 | 31.8 | 35.2 | 29.1 | 30.6 | 34.8 | 29.3 | 33.7 | 39.2 | 26.9 |

• VRel = visual relation
• Count = counting
• Attr = attribute
• Depth = depth & 3D perception
• Loc = localization
• Comp = comparison
• FGR = fine-grained recognition
• Ctx = contextual integration
• OCR = optical character recognition
• Hallu = perception-related hallucination

Quick Start: Evaluate Your Model

1. Install

git clone git@github.com:MoonshotAI/PerceptionBench.git
cd PerceptionBench
pip install -r requirements.txt

2. Configure

cp .env.example .env # then fill in your credentials (auto-loaded via python-dotenv)

or export the variables directly:

export OPENAI_API_KEY="your-api-key-here"
export OPENAI_BASE_URL="https://your-endpoint/v1" # any OpenAI-compatible endpoint

3. Run

python eval/eval.py

All configuration is read from environment variables (or .env):

| Variable | Default | Description | |----------|---------|-------------| | OPENAI_API_KEY | required | API key for the OpenAI-compatible endpoint | | OPENAI_BASE_URL | required | Base URL of the OpenAI-compatible endpoint | | MODEL | required | Model under evaluation | | JUDGE_MODEL | gpt-oss-120b | LLM judge for free-form answers (matches the paper) | | DATASET | PerceptionBench.jsonl | Path to the benchmark JSONL file | | CONCURRENCY | 16 | Concurrent requests | | MAX_TOKENS | 65536 | Max generation tokens per question | | MAX_RETRIES | 3 | API-call retries before giving up; -1 = retry forever | | PRED_TEMPERATURE | unset | Prediction sampling temperature (judge runs at 0.3) | | TOP_P | unset | Nucleus sampling for predictions | | TOP_K | unset | Top-k sampling for predictions (via extra_body) |

Per-item predictions are written to...