NVIDIA/instant-nurec

NVIDIA/instant-nurec

Description: InstantNuRec: Feed-Forward 3D Gaussian Reconstruction from Driving Logs

Language: Python

License: Apache-2.0

Stars: 17

Forks: 0

Open issues: 0

Created: 2026-05-22T18:08:59Z

Pushed: 2026-05-29T13:39:42Z

Default branch: main

Fork: no

Archived: no

README:

InstantNuRec: Feed-Forward 3D Gaussian Reconstruction from Driving Logs

NVIDIA

Abstract

Reconstructing dynamic outdoor scenes from autonomous-vehicle driving logs traditionally requires lengthy per-scene optimization. InstantNuRec takes a different route: a feed-forward transformer directly infers a dynamic 3D-Gaussian scene representation in a single forward pass. Given a short window of multi-camera observations from an AV log, the model emits a Gaussian primitive per pixel — covering geometry, appearance, and per-Gaussian motion — which can be rendered in real time and interchanged with existing simulation pipelines.

This repo goes from ncorev4 ingest → frame batch prep → forward pass → 3D-Gaussian PLY export. The PLY output is usable directly as a static reconstruction, and can also serve as initialization for downstream NuRec training to reach higher fidelity.

Instant-NuRec and NuRec share the same input (NCore V4 clip / HF dataset / sequence .json) but run on different runtimes: Instant-NuRec is a native-Python feed-forward preview (seconds per clip); NuRec is a Docker-based per-scene refinement pipeline that produces a high-fidelity USDZ.


Background

Instant NuRec is a feed-forward reconstruction model that converts driving logs into 3D Gaussian Splatting (3DGS) representations. Its vision-transformer backbone and DPT-decoders output a high-fidelity 3D environment that's ready for simulations.

Instant NuRec leverages the following foundational technologies: Depth-Anything-V3, STORM, and BTimer.

Pipeline Overview

NCore V4 Sequence ─► Frame Batching ─► Forward Pass (JIT) ─► 3D Gaussians ─► PLY (per-chunk or merged)

User Guide

Setup

Prerequisites

• Python 3.11
• NVIDIA driver and GPU VRAM — see the

NuRec Hardware Setup and Requirements page; Instant-NuRec inherits the same minimums.

• uv — the Astral Python package manager.

Install with curl -LsSf https://astral.sh/uv/install.sh | sh or pip install uv.

git clone https://github.com/NVIDIA/instant-nurec.git
cd instant-nurec
./setup.sh
source .venv/bin/activate

setup.sh runs uv sync --frozen, which installs the locked dependency tree from uv.lock into .venv/. The only CUDA dependency is whatever the pinned torch wheel ships with.

This repo is native-Python only — no Docker required. If you want a container, use the standard NuRec image as a generic CUDA environment.

Download Model Checkpoints [optional]

> Note: instant_nurec.pt is auto-downloaded into the Hugging Face > hub cache on the first inference run.

However, you can also manually download the model into a directory of your choice:

pip install huggingface_hub[cli]
hf auth login
hf download nvidia/instant-nurec --local-dir checkpoints

This places the following file in checkpoints/:

checkpoints/ └── instant_nurec.pt

Point the pipeline at this local copy by exporting:

export INSTANT_NUREC_FULL_PT="$(pwd)/checkpoints/instant_nurec.pt"

Inference

> Note: The pretrained model instant_nurec.pt (a TorchScript > archive) is fetched on first inference run from the Hugging Face > repo nvidia/instant-nurec and cached locally; subsequent runs read > it from the cache. Set INSTANT_NUREC_FULL_PT to a local path to > override the auto-download.

##### First run — end-to-end on a public demo clip

The clip lives in a gated HF dataset. Accept the terms at nvidia/PhysicalAI-Autonomous-Vehicles-NCore while logged into Hugging Face, then hf auth login locally; the same auth covers the nvidia/instant-nurec model auto-download on first run.

# Download the clip (~2 GB)
hf download \
nvidia/PhysicalAI-Autonomous-Vehicles-NCore --repo-type dataset \
--include "clips/000da9de-0ee5-465a-9a2d-e7e91d3016bb/*" \
--local-dir ./demo_clip

# Reconstruct it
python run_inference.py \
--ncore-path ./demo_clip/clips/000da9de-0ee5-465a-9a2d-e7e91d3016bb/pai_000da9de-0ee5-465a-9a2d-e7e91d3016bb.json \
--output-dir ./demo_output \
--merge

Success looks like a single PLY at ./demo_output//ply/pai_000da9de-.../pai_000da9de-....ply — ~1.88 M Gaussians, kl-optimal voxelized from 2.87 M merged (3.18 M pre-merge across 2 chunks) to land in [0.9 * --n-gaussians, --n-gaussians] (default target 2 M). Omit --merge to write per-chunk PLYs instead (voxelization is bundled with merge and runs only when the flag is set).

##### View your output

The PLY is a 3DGS PLY (Gaussian Splatting), not a point cloud — generic viewers like MeshLab / macOS Preview will fail to open it. Use one of:

• SuperSplat — browser, no install.
• ply_viewer — shipped in the NuRec container.

--ncore-path accepts two input shapes:

##### Mode 1 — single sequence .json (NuRec-aligned)

The path is treated as one ncorev4 sequence metadata file. This matches NuRec's own input convention.

./run.sh \
--ncore-path /path/to/clips//pai_.json \
--output-dir /tmp/out

##### Mode 2 — .lst manifest (batch)

The path is treated as a list of sequence JSON paths, one per line. Each line may be absolute, relative-to-the-LST-file's directory, or ~/-prefixed; lines starting with # and blank lines are skipped; mixed absolute + relative entries in a single LST are supported.

# example_manifest.lst
/abs/path/to/clips//pai_.json
relative/path/to/clips//pai_.json
~/symlinked/clips//pai_.json

./run.sh \
--ncore-path /path/to/example_manifest.lst \
--output-dir /tmp/out \
--merge

run.sh validates the input + output paths and execs python run_inference.py. You can also call the CLI directly: