nvidia/Cosmos3-H-Embodied-Nano

Cosmos3-H-Embodied-Nano

Description

Cosmos3-H-Embodied-Nano is Cosmos3-Nano (16B) post-trained for 50,000 iterations on the Open-H-Embodiment surgical robotics data (15 dataset families) jointly for forward dynamics, inverse dynamics and policy, with up to 16 past actions as clean conditioning. It is a single Mixture-of-Transformers (MoT) that couples video, action, and text in one multimodal sequence. One model predicts robot actions and endoscopic video for dVRK-style and other surgical robots on 2.4 s clips (25 frames and 24 actions at 10 Hz):

• Forward dynamics: first frame and actions → video.
• Inverse dynamics: video → actions.
• Policy: first frame, text (task instruction) and action history → actions and video.

The three tasks are mixed 0.4 / 0.2 / 0.4 (forward dynamics / inverse dynamics / policy) during training. Because they are conditioning patterns of one shared model, a single checkpoint learns both directions of the surgical world–action relationship. It can be used directly in any of the three modes or fine-tuned further, for example on a single mode or on a new robot. For more information on the base model, please view the Cosmos3-Nano Model Card.

Cosmos3-H-Embodied-Nano is the successor of Cosmos-H-Surgical-Simulator, which was a forward-dynamics-only surgical simulator built on the 2B-parameter Cosmos-Predict2.5 diffusion transformer. Cosmos3-H-Embodied-Nano moves to the larger Cosmos3 MoT architecture and to joint training of forward dynamics, inverse dynamics and policy.

The model is intended for surgical world-model generation, closed-loop evaluation and imitation-learning of surgical robotics policies, video-to-kinematics (inverse-dynamics) action labeling, and synthetic-data generation prior to deployment on a physical system.

This model is ready for commercial or non-commercial use.

License/Terms of Use

GOVERNING DOWNLOAD TERMS: Use of the model is governed by the NVIDIA Open Model License Agreement.

Additional Information: Use of the Cosmos3-Nano base model is governed by the NVIDIA Open Model License Agreement.

Deployment Geography

Global

Use Case

Primarily intended for surgical robotics researchers, healthcare AI developers, academic institutions, and surgical robotics companies exploring surgical world modeling, closed-loop surgical policy evaluation, imitation learning, inverse-dynamics action labeling of surgical video, and synthetic data generation. It can be served directly in any of its three modes (including through the HTTP policy server) or used as the initialization for fine-tuning on a single mode or on new surgical robot data.

Release Date

• GitHub: 10/20/2026 via https://github.com/NVIDIA-Medtech/Cosmos3-H-Embodied-Nano
• Huggingface: 10/20/2026 via https://huggingface.co/nvidia/Cosmos3-H-Embodied-Nano

Reference(s)

NVIDIA Cosmos Team. (2026). Cosmos 3: Omnimodal World Models for physical AI [Technical report]. NVIDIA. https://research.nvidia.com/labs/cosmos-lab/cosmos3/technical-report.pdf

@techreport{nvidia2026cosmos3,
title = {Cosmos 3: Omnimodal World Models for physical AI},
author = {{NVIDIA Cosmos Team}},
year = {2026},
url = {https://research.nvidia.com/labs/cosmos-lab/cosmos3/technical-report.pdf}
}

Source code: NVIDIA-Medtech/Cosmos3-H-Embodied-Nano (plugin package cosmos3_h_embodied on cosmos-framework)

Base model: nvidia/Cosmos3-Nano

Dataset: nvidia/PhysicalAI-Robotics-Open-H-Embodiment

Cosmos-H-Surgical-Simulator (forward-dynamics-only predecessor model): nvidia/Cosmos-H-Surgical-Simulator, described in: Ali, A., Bai, J., Bala, M., Balaji, Y., Blakeman, A., Cai, T., Cao, J., Cao, T., Cha, E., Chao, Y.-W., Chattopadhyay, P., Chen, M., Chen, Y., Cheng, S., Cui, Y., Diamond, J., Ding, Y., Fan, J., Fan, L., Feng, L., Ferroni, F., Fidler, S., Fu, X., Gao, R., Ge, Y., Gu, J., … Zhu, Y. (2025). World Simulation with Video Foundation Models for Physical AI (arXiv:2511.00062) [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2511.00062

Model Architecture

Architecture Type: Mixture-of-Transformers (MoT) Network Architecture: Unified world foundation model that couples a Reasoner (vision-language) tower and a Generator (world-simulator / diffusion) tower which share multimodal attention over one interleaved sequence of video, action, and text tokens. Actions are consumed and (in inverse-dynamics / policy modes) predicted through dedicated, embodiment-specific action projections.

Number of Model Parameters: 16 billion (the official Cosmos3-Nano size; see nvidia/Cosmos3-Nano). The Cosmos3-Nano MoT couples a Qwen3-VL-8B reasoning tower with a diffusion generation expert. Post-training updates the diffusion generation expert, the timestep embedder, the video-latent and action input/output projections, and the action modality embedding (approximately 7 billion parameters); the reasoning tower is frozen.

This model was developed based on Cosmos3-Nano.

Cosmos3-H-Embodied-Nano post-trains Cosmos3-Nano, a world foundation model whose backbone is a Qwen3-VL-8B reasoning tower plus a diffusion generation expert (a Mixture-of-Transformers that natively supports text2image, text2video, image2video, and the action modes forward dynamics, inverse dynamics, and policy). Video is tokenized in latent space with the Wan2.2 video tokenizer (spatial compression 16×, temporal compression 4×, 2×2 spatial patchification). Actions enter and exit the shared sequence through domain-aware linear projections: action2llm embeds the action vector into the model's token space, llm2action is the action-prediction head trained by the inverse-dynamics and policy modes, and a...