NVIDIA/Spatial-IQ

NVIDIA/Spatial-IQ

Description: A diagnostic framework that decomposes 3D object counting into nine hierarchical spatial perception and cognition sub-tasks. Analysis code for the paper Spatial-IQ: Deconstructing Spatial Intelligence via Hierarchical Capability Tests.

Language: Python

License: BSD-3-Clause

Stars: 1

Forks: 0

Open issues: 0

Created: 2026-06-22T14:44:07Z

Pushed: 2026-07-23T17:33:30Z

Default branch: main

Fork: no

Archived: no

README:

Spatial-IQ Analyses

Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. Licensed under the [BSD-3-Clause](LICENSE) License.

Analysis code for the paper Spatial-IQ: Deconstructing Spatial Intelligence via Hierarchical Capability Tests.

🌐 [Project Page →](https://nvidia.github.io/Spatial-IQ/) | 📊 [Dataset (HuggingFace) →](https://huggingface.co/datasets/patrickqrim/spatial-iq)

The main entry point is spatial-IQ_analyses.py. It reads model inference-result CSVs and generates the paper-aligned analysis tables, statistics, and figures used for the manuscript and appendix.

What This Script Does

The analysis pipeline has three stages:

1. Discover raw model-result CSVs in inference_results/. 2. Export analysis-ready CSV tables with paper figure/table identifiers. 3. Render paper figures from those exported CSV tables.

This table-first design is meant for appendix review: each generated figure has a corresponding CSV with counts, uncertainty estimates, and statistical annotations where applicable.

Repository Layout

Spatial-IQ/
├── spatial-IQ_analyses.py # Paper-aligned analysis and figure script
├── inference_results/ # Raw eval_raw_*.csv inputs
├── analyses_results/ # Generated paper tables, figures, and manifest
├── docs/ # Sphinx source for the project page (GitHub Pages)
│ ├── conf.py
│ ├── index.md # Project landing page
│ └── _static/ # Custom CSS and figures for the page
├── scripts/
│ └── make_demo.py # Regenerates docs/_static/figures/demo.gif
├── demo_gt/ # Ground-truth scene inputs to make_demo.py
├── demo_gemini/ # Gemini prediction inputs to make_demo.py
├── pyproject.toml # uv-managed docs build dependencies
├── uv.lock
├── .github/workflows/pages.yml # GitHub Pages auto-deploy workflow
├── LICENSE # BSD-3-Clause license
├── THIRD_PARTY_LICENSES.md # Third-party dependency notices
└── README.md

Project Page

The Spatial-IQ project page is live at [nvidia.github.io/Spatial-IQ](https://nvidia.github.io/Spatial-IQ/).

It is built with Sphinx and MyST-Markdown from the docs/ folder, and deployed automatically to GitHub Pages via [.github/workflows/pages.yml](.github/workflows/pages.yml) on every push to main.

To build the page locally:

# Install docs build dependencies (uv-managed)
uv sync

# Build to docs/_build/html/
uv run sphinx-build -b html docs docs/_build/html

# Serve locally
uv run python -m http.server -d docs/_build/html 8765
# then visit http://127.0.0.1:8765/

The page copy lives in [docs/index.md](docs/index.md); shared figures under [docs/_static/figures/](docs/_static/figures/); custom CSS in [docs/_static/spatial_iq.css](docs/_static/spatial_iq.css). To regenerate the animated demo GIF, run uv run python scripts/make_demo.py.

Dataset

The Spatial-IQ benchmark dataset is available on Hugging Face:

https://huggingface.co/datasets/patrickqrim/spatial-iq

Quick Start

Install the Python dependencies:

pip install pandas numpy scipy matplotlib

Run the full analysis pipeline:

python spatial-IQ_analyses.py --stage all

Generated files are written to analyses_results/ by default:

analyses_results/
├── paper_output_manifest.csv
├── paper_output_manifest.json
├── tables/
└── figures/

Using New Model Outputs

This repository does not run model inference by itself. It assumes that inference has already been run on the Spatial-IQ benchmark examples and that the results have been saved in the CSV structure described below.

To evaluate a new model, use the Spatial-IQ dataset at https://huggingface.co/datasets/patrickqrim/spatial-iq, ask the model the paper prompts for each task, score the model responses against the provided ground truth, and save the scored results as eval_raw_*.csv files in inference_results/.

The paper prompt specification is in Appendix A, “Prompts, Definitions, and Task Specifications.” The appendix describes the shared prompt structure, concept definitions, modality-specific wording, and wrong-answer taxonomy. The task-specific query text is also summarized in Table 5 of the paper and exported by this script as analyses_results/tables/table_05_task_queries.csv.

Once a model's inference results are converted to the expected CSV format, the same analysis command can be used for any model:

python spatial-IQ_analyses.py --stage all

CLI Options

python spatial-IQ_analyses.py \
--input-dir inference_results \
--output-dir analyses_results \
--stage all \
--groups text mcq5 mcq345 image trained_text

--stage can be all, analysis, figures, or manifest.

--groups controls which modalities are processed. Supported groups are text, mcq5, mcq345, image, and trained_text.

Input CSV Discovery

Model lists are not hardcoded. The script discovers inputs by filename pattern:

| Group | Expected inputs | | --- | --- | | text | eval_raw_*_text.csv and eval_raw_human_frq.csv, excluding trained checkpoints | | mcq5 | eval_raw_*_mcq.csv, excluding 3CQ and 4CQ variants | | mcq345 | all eval_raw_*_mcq.csv files | | image | eval_raw_image_*.csv files | | trained_text | trained checkpoint text CSVs, including names such as dapo*, sft-*, and Qwen2.5-VL* |

Adding a new model usually only requires placing the matching CSV in inference_results/ and rerunning the pipeline.

Input CSV Structure

Each CSV should contain one row per evaluated scene/request. Task correctness columns should be numeric binary values, where 1 means correct and 0 means incorrect.

Core task columns:

| Column | Meaning | | --- | --- | | task1 ... task11 | Binary correctness for each Spatial-IQ sub-task/target column used by the raw evaluation code. | | main | Binary correctness for Object Counting. Image-output CSVs may use task_main; the script renames it to main. |

Recommended metadata columns for reproducing all paper analyses:

| Column |...