nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-Base-BF16

NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-Base-BF16

> *Looking for a post-trained model? See the BF16 release for customization or the NVFP4 release for optimized deployment.*

Model Overview

Model Developer: NVIDIA Corporation

Description

NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-Base-BF16 is the base vision-language checkpoint of the Nemotron 3.5 Super family. It provides a starting point for continued training, domain adaptation, and developing custom post-trained vision-language models.

The language backbone is initialized from Nemotron 3 Super Base and further trained through distillation from Nemotron 3 Ultra Base. The model underwent continued pre-training on indic languages, long-context training, and software-engineering mid-training. A vision encoder and connector extend the language backbone to visual inputs.

The model combines interleaved Mamba-2, Mixture-of-Experts (MoE), and attention layers with a C-RADIO vision encoder and a vision-language connector. It has 120B total parameters and 12B active parameters and supports context lengths up to 1M tokens. Multi-Token Prediction (MTP) enables native speculative decoding.

This model is ready for commercial or non-commercial use.

What is Nemotron?

NVIDIA Nemotron™ is a family of open models with open weights, training data, and recipes, delivering leading efficiency and accuracy for building specialized AI agents.

License/Terms of Use

Governing Download Terms: Use of this model is governed by the OpenMDW License Agreement, version 1.1.

Benchmarks

The following results evaluate the Nemotron 3.5 Super text-only base backbone, before vision-language training.

| Benchmark | Metric | Score (%) | |:---|:---|---:| | MMLU | Exact match | 84.14 | | MMLU-Pro | Exact match, TigerLab extraction | 76.25 | | AGIEval-En | CoT exact match | 75.65 | | SuperGPQA | CoT | 47.65 | | GPQA Diamond | 5-shot, CoT exact match | 66.16 | | GSM8K | 8-shot, CoT exact match | 91.74 | | Minerva MATH | Exact match | 87.66 | | MATH-500 | 4-shot, sampled pass@1, n=32 | 81.56 | | HumanEval | EvalPlus sanitized, sampled pass@1, n=32 | 81.31 | | MBPP-Sanitized | 3-shot, sampled pass@1, n=32 | 79.06 | | LiveCodeBench CodeGen | Pass@1 | 47.61 | | BigCodeBench Complete | Greedy | 48.42 | | BigCodeBench Instruct | Greedy | 49.39 | | CommonsenseQA | 7-shot, accuracy | 85.26 | | ARC-Challenge | 25-shot, normalized accuracy | 95.99 | | HellaSwag | 10-shot, normalized accuracy | 88.96 | | OpenBookQA | 0-shot, normalized accuracy | 48.80 | | PIQA | 0-shot, normalized accuracy | 85.09 | | TruthfulQA MC2 | 0-shot, accuracy | 71.18 | | WinoGrande | 5-shot, accuracy | 79.08 | | SocialIQA | 0-shot, accuracy | 49.03 | | RACE | 0-shot, accuracy | 91.39 | | Global MMLU | Average across 8 languages | 86.56 | | MGSM | 8-shot, native CoT, average across 6 languages | 87.53 |

Deployment Geography: Global

Use Case

This model is intended for developers and researchers building custom vision-language models through continued pre-training, supervised fine-tuning, reinforcement learning, and distillation.

Release Date

Hugging Face - 10/15/2026 via Hugging Face

Reference(s)

• NVIDIA Nemotron 3 model family on Hugging Face
• NVIDIA Nemotron 3.5 Super VL and Lightning Technical Report
• NVIDIA Nemotron 3 Ultra Technical Report

Model Architecture

• Architecture Type: Mamba-2–Transformer hybrid Mixture of Experts (MoE) with Multi-Token Prediction (MTP)
• Network Architecture: Nemotron Hybrid MoE with 512 routed experts per MoE layer, a C-RADIO vision encoder, and a vision-language connector
• Number of Model Parameters: 120B total / 12B active

Model Design

The model combines sparse expert computation, attention, and Mamba-2 layers to support long-context processing. The vision encoder and connector turn visual inputs into representations processed by the language backbone. A shared MTP head predicts future tokens recursively for speculative decoding.

The model supports up to 1M tokens of context. The prompt, visual inputs, conversation history, tool results, and generated response share this context window. Individual deployment recipes may use a smaller context window to accommodate the available GPU memory and concurrency.

Training Methodology

Stage 1: Foundation and Multi-Token Prediction

The language backbone is initialized from Nemotron 3 Super Base. A shared MTP head is trained for 100B tokens while the backbone remains frozen, followed by 1T tokens of distillation from Nemotron 3 Ultra Base.

Stage 2: Continued Pre-Training and Mid-Training

The model undergoes 100B tokens of Indic continued pre-training, a 35B-token extension to 1M context, 67B tokens of software-engineering mid-training at 256K context, and a final 9B-token long-context recovery stage. Software-engineering data includes repositories, issue–pull-request pairs, commit diffs, build and CI logs, and agent trajectories.

Stage 3: Vision-Language Training

A C-RADIO vision encoder and connector extend the language backbone to visual inputs, including images, document pages, and video frames.

Input

• Input Type(s): Text, images, and video
• Input Format(s): Text strings, images, and sampled video frames
• Video: mp4, up to 2 minutes.
• Image: Red, Green, Blue (RGB) (jpeg, png)
• Input Parameters:
• Video: Three-Dimensional (3D)
• Image: Two-Dimensional (2D)
• Text: One-Dimensional (1D)
• Other Properties Related to Input: Maximum context length up to 1M tokens. Documents can be supplied as page images with accompanying text.

Output

• Output Type(s): Text
• Output Format: String
• Output Parameters: One-dimensional sequences
• Other Properties Related to Output: Maximum context length up to 1M tokens.

Our AI models are designed and optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU...