Tencent-Hunyuan/VisualNeedle

Tencent-Hunyuan/VisualNeedle

Description: A benchmark for active visual search in high-information-density scenes

Language: Python

License: NOASSERTION

Stars: 0

Forks: 0

Open issues: 0

Created: 2026-08-05T08:12:16Z

Pushed: 2026-08-05T08:20:36Z

Default branch: main

Fork: no

Archived: no

README:

VisualNeedle Evaluation Harness

Evaluation code for VisualNeedle, a benchmark for active visual search in high-information-density scenes. The harness supports three evaluation modes, concurrent execution, a VLM-based semantic judge, and interactive HTML reports.

---

Contents

run_visualneedle_eval.py # Standard evaluation (tool-enabled or no-tool)
run_crop_black_ablation.py # Ablation: tool images replaced with black
run_text_only_eval.py # Baseline: no image input, text-only judge
generate_html_report.py # HTML report generator

visualneedle_eval/ # Core package
├── runner.py # Per-sample evaluation pipeline
├── eval_loop.py # Async / multiprocessing scheduler
├── visualneedle_agent.py # Model config loading, agent construction
├── matching.py # Answer matching and normalisation
├── judge.py # VLM semantic judge
├── artifacts.py # Result structures, logging, statistics
├── image_io.py # Image I/O and resizing
├── image_registry.py # Image provenance tracking with affine matrices
├── bbox_geometry.py # Coordinate transforms
├── report.py # HTML report rendering
├── dataset.py # Dataset loading and sampling
├── cli.py # Shared model-config logic for entry scripts
├── _worker.py # Multiprocessing worker state
├── _retry.py # Retry logic for incomplete runs
└── _types.py # EvalMode type definition

configs/models/ # YAML model configuration files
Qwen-Agent/ # Vendored Qwen-Agent runtime

---

Installation

# Runtime dependencies (includes the vendored Qwen-Agent)
pip install -r requirements.txt
pip install -e .

For development and tests:

pip install -r requirements-dev.txt
make test

If you prefer not to install the package, prefix every command with:

PYTHONPATH=Qwen-Agent:. python .py ...

---

Model Configuration

All model and judge settings are declared in a YAML config file. A starter config is provided at:

configs/models/think_with_images_neibu.yaml

The config file location is resolved in this order:

1. --config /path/to/config.yaml (CLI flag) 2. VISUALNEEDLE_MODEL_CONFIG=/path/to/config.yaml (environment variable) 3. The default path compiled into visualneedle_agent.py

Config structure (minimal example)

defaults:
active_model: my-model # used when --model is not specified
judge_model: my-judge-model # used when --judge-model is not specified

models:
my-model:
model_type: oai # oai | claude | qwenvl_oai | ...
model: gpt-4o
model_server: https://api.openai.com/v1
api_key: ${OPENAI_API_KEY} # or api_key_env: OPENAI_API_KEY
tools:
image_zoom_in_tool_reason:
bbox_order: xyxy
bbox_scale: 1000

my-judge-model:
model_type: oai
model: gpt-4o-mini
model_server: https://api.openai.com/v1
api_key: ${OPENAI_API_KEY}

Common config flags (shared by all three evaluation scripts):

| Flag | Default | Description | |---|---|---| | --config FILE | env / built-in | YAML model config file | | --model NAME | defaults.active_model | Model name from the YAML models section | | --judge-model NAME | defaults.judge_model | Judge model name |

---

Evaluation Modes

Mode 1 — Standard (run_visualneedle_eval.py)

Full VQA evaluation. The model receives the image and (optionally) calls image tools such as zoom, crop, flip, and sharpen to inspect details before answering.

# Tool-enabled (default)
python run_visualneedle_eval.py \
--config configs/models/think_with_images_neibu.yaml \
--model my-model \
--dataset-file data/visualneedle.jsonl \
--concurrency 20

# No-tool: direct VL evaluation, single turn
python run_visualneedle_eval.py \
--config configs/models/think_with_images_neibu.yaml \
--model my-model \
--dataset-file data/visualneedle.jsonl \
--no-tools \
--concurrency 20

Mode 2 — Crop-Black Ablation (run_crop_black_ablation.py)

Identical to standard mode, except every image returned by a tool is replaced with a same-size black image before being shown to the model. Used to measure how much the model relies on tool-output visual content rather than text reasoning.

python run_crop_black_ablation.py \
--config configs/models/think_with_images_neibu.yaml \
--model my-model \
--dataset-file data/visualneedle.jsonl \
--concurrency 20

Mode 3 — Text-Only Baseline (run_text_only_eval.py)

No image is passed to the model. The prompt tells the model it has no image access and asks for its best guess from text alone. Uses a text-only judge (no image). Establishes the prior-knowledge baseline.

python run_text_only_eval.py \
--config configs/models/think_with_images_neibu.yaml \
--model my-model \
--dataset-file data/visualneedle.jsonl \
--concurrency 20

> --no-tools and --data-format are accepted for CLI compatibility but are always ignored in text-only mode.

---

All CLI Flags

The flags below apply to all three evaluation scripts unless noted.

Dataset selection

| Flag | Default | Description | |---|---|---| | --dataset-file FILE | built-in default | Path to a JSONL dataset | | --limit N | — | Evaluate at most N samples (smoke test) | | --start N | 0 | Start index (ignored when --indices is set) | | --indices 0,12,99 | — | Evaluate specific indices only; overrides --start/--limit |

Concurrency

Two concurrency modes are mutually exclusive. Use one or the other:

| Flag | Default | Description | |---|---|---| | --concurrency N | 0 (disabled) | Async I/O mode — recommended for API-bound evaluation. N = max concurrent requests (20–50 is typical) | | --workers N | 1 | Multiprocessing mode — spawns N worker processes. Use when you need process isolation |

When both are set, --concurrency takes precedence.

Logging

| Flag | Default | Description | |---|---|---| | --log-dir DIR | — | Directory for per-sample logs and structured results | | --auto-log-dir / --no-auto-log-dir | enabled | Auto-generate a timestamped log directory. Disable for maximum speed | | --verbose | off | Print full per-sample logs to stdout (very slow with concurrency > 1) | | --ordered-output / --no-ordered-output | auto (TTY-detect) | Print results in dataset-index order when piping output...