nvidia/NL-Diffusion-Image

Model Overview

Description

NL-Diffusion-Image introduces a new paradigm for high-resolution text-to-image generation via LLM based on masked discrete diffusion over tokenized image patches. Each image is encoded into a sequence of discrete tokens (using a 128K codebook/vocabulary), and generation proceeds through iterative parallel unmasking - similar to Diffusion LLMs. We finetune from Nemotron-Labs-Diffusion and introduce 2 key components:

• A token-editing mechanism that allows the model to revise already-unmasked tokens during inference.
• Grouped Cross-Entropy (GCE) objective to handle large-vocabulary training efficiently.

This model is for research and development only.


License/Terms of Use

GOVERNING TERMS: Use of this model is governed by the NVIDIA Open Source License Agreement.

Deployment Geography

Global

Use Case

This model is intended for individuals who would like to research and evaluate text-to-image generation.

Release Date

Hugging Face: 07/01/2026 via HuggingFace.

References

• NL-Diffusion-Image Paper: Shufan Li et al., "Nemotron-Labs-Diffusion-Image: Advancing Masked Discrete Diffusion for High-Resolution Image Synthesis,".
• Nemotron-Labs-Diffusion Paper: Yonggan Fu et al., "Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding".
• Emu3.5 Paper: Emu3.5 team, "Emu3.5: Native Multimodal Models are World Learners".

Model Architecture

Architecture Type: Neural Network

Network Architecture: Masked Diffusion Transformer, IBQ tokenizer for visual encoding/decoding

Number of model parameters: ~8B parameters

We encode 16x16 image patches using a pretrained discrete tokenizer from Emu3.5, with a codebook size of 128k token IDs. We expand the Nemotron-Labs-Diffusion vocabulary with a corresponding number of randomly-initialized embeddings, and fine-tune the model on a dataset of image/caption pairs.

Computational Load (Internal Only: For NVIDIA Models Only)

Estimated using Compute ≈ 6ND, with N the number of trained parameters (encoder + training-only decoder ≈ 1.45B) and D the number of patch tokens processed during training. With 58B tokens for pre-training, 255B tokens for Supervised Fine Tuning, D ≈ 3.1×10¹¹ tokens (image and text included).

Cumulative Compute: ≈ 1.5e22 FLOPS.

Estimated Energy: ≈ 53000 kWh.

Estimated Emissions for Model Training: ≈ 22.4 tCO2e.

Input

Input Type(s): Text

Input Format(s): Character Strings

Input Parameters: One-dimensional (1D)

Other Properties Related to Input: Maximum prompt length is 900 tokens.

Output

Output Type(s): Images

Output Format: Tensor (3xHxW)

Output Parameters: Two-Dimensional (2D)

Other Properties Related to Output: The output represents an RGB image.

Software Integration

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Runtime Engine(s):

• PyTorch

Supported Hardware Microarchitecture Compatibility:

• NVIDIA Ampere

• NVIDIA Blackwell

• NVIDIA Jetson

• NVIDIA Hopper

• NVIDIA Lovelace

• NVIDIA Pascal

• NVIDIA Turing

• NVIDIA Volta

Supported Operating System(s):

• Linux
• Linux 4 Tegra
• QNX
• Windows

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

This AI model can be embedded as an Application Programming Interface (API) call into the software environment described above.

Model Version(s)

• NL-Diffusion-Image (8B).

Links:

• https://huggingface.co/nvidia/NL-Diffusion-Image

Training and Evaluation Datasets

Training Dataset

LAION-115M-Clean Recaptioned

A large, filtered collection of diverse real-world web images - including people, animals, objects, scenes, artwork, products, documents, and memes - paired with high-quality synthetic text captions generated from the images.

Data Modality:

• Multimodal (image, text)

Image Training Data Size:

• 115M samples

Data Collection Method by dataset:

• Automated

Labeling Method by dataset:

• Synthetic: Subset of 8M images recaptioned using Qwen3-VL

MidJourney v6 520k Recaptioned

High-quality synthetic Midjourney v6 images from user prompts.

Data Modality:

• Multimodal (image, text)

Image Training Data Size:

• 520k samples

Data Collection Method by dataset:

• Automated

Labeling Method by dataset:

• Synthetic: Images recaptioned using Qwen3-VL

COYO700M Recaptioned

Image–alt-text pairs from HTML pages.

Data Modality:

• Multimodal (image, text)

Image Training Data Size:

• 700M samples

Data Collection Method by dataset:

• Automated

Labeling Method by dataset:

• Synthetic: Subset of 24M images recaptioned using Qwen3-VL

BLIP3o-60k Recaptioned

Data Modality:

• Multimodal (image, text)

Image Training Data Size:

• 520k samples

Data Collection Method by dataset:

• Automated

Labeling Method by dataset:

• Synthetic: Images recaptioned using Qwen3-VL

Evaluation Datasets

ImageNet

Labeled natural images of objects and scenes organized by WordNet categories.

Link:

• ImageNet

Data Modality:

• Images

Data Collection:

• Automated

Labeling Method:

• Manually-Collected

Training Images:

• 1,281,167

Validation Images:

• 50,000

GenEval

Prompts for testing objects, attributes, counts, colors, and spatial...