tencent/Hy-Embodied-VLM-1.0

🔥 Updates

• `[2026-07-15]` 🚀 We have released Hy-Embodied-VLM-1.0! An efficient Mixture-of-Experts vision–language foundation model for embodied agents in the physical world, activating only ~3B parameters per token (~30B total) for high inference efficiency. Weights are available on Hugging Face, together with inference code for both HuggingFace transformers and vLLM.
• `[2026-06-15]` 🤖 We have released HY-VLA-0.5! The official code, UMI-trained weights and 2000+ hours of high-fidelity UMI data are now available.
• `[2026-04-09]` 🚀 We have released HY-Embodied-0.5, featuring the open-sourced HY-Embodied-0.5 MoT-2B weights on Hugging Face along with the official inference code!

📖 Abstract

Building capable embodied agents requires not only multimodal perception and understanding, but also agentic capabilities for reasoning about actions, adapting to evolving situations, and interacting with the physical world. In this report, we introduce Hy-Embodied-VLM-1.0, an efficient and powerful embodied foundation model specifically designed for embodied agents operating in the physical world.

To cultivate such capabilities from the pre-training stage onward, we define an action-centric capability taxonomy comprising three progressive dimensions: Action-Relevant State Understanding, Action–Transition Reasoning, and Sequential and Adaptive Reasoning. Guided by this taxonomy, we develop a systematic data pipeline and curate data mixtures spanning both pre-training and post-training.

To deliver strong physical-world understanding and interaction capabilities while supporting latency-sensitive deployment, we build our model on the Hy3-A3B language backbone and the Hy-ViT2 vision encoder. Its efficient Mixture-of-Experts architecture combines strong model capacity with high inference efficiency. We evaluate Hy-Embodied-VLM-1.0 on a comprehensive suite of 38 benchmarks covering embodied perception, physical-world understanding, and embodied reasoning. The model achieves the best performance among similarly sized models on 19 of the 38 benchmarks and substantially outperforms strong competitors, including Qwen3.6-A3B and Cosmos 3. Compared with the previous-generation Hy-Embodied-0.5 MoT-2B, Hy-Embodied-VLM-1.0 improves average performance by 8.4%. Despite activating only 3B parameters, it achieves performance close to that of the previous-generation model with 32B activated parameters. Beyond static benchmark evaluation, Hy-Embodied-VLM-1.0 also demonstrates strong performance on embodied agentic tasks requiring multi-turn interaction and long-horizon reasoning.

⭐️ Key Features

• 🧠 Efficient MoE, ~3B activated — Combines the Hy3-A3B language backbone with the Hy-ViT2 vision encoder in a Mixture-of-Experts architecture. Only ~3B parameters are activated per token — approximately one-tenth of the activated parameters of the previous-generation A32B system, while achieving nearly comparable overall performance.
• 🌏 Action-Centric Capability Taxonomy — We define three progressive levels of embodied intelligence: (i) Action-Relevant State Understanding for accurately understanding the states of the agent and its environment, (ii) Action–Transition Reasoning for understanding actions, planning them, and reasoning about their consequences, and (iii) Sequential and Adaptive Reasoning for long-horizon planning, reflection, repair, and recovery. Data and training are systematically designed around this taxonomy.
• 🔁 Self-Evolving Post-Training — Embodied agentic reasoning is cultivated through a self-evolving loop that couples reinforcement learning with rejection-sampling fine-tuning, seeded from a small curated set of high-quality thinking traces. A final reward-specialized stage trains continuous-reward and discrete-reward RL policies separately and fuses them, delivering sharp geometric precision alongside robust decision-making, planning, and reflection quality.
• 🏆 State-of-the-Art on Embodied Benchmarks — Ranks 1st on 19 of 38 benchmarks and 2nd on another 11, outperforming Qwen3.6-A3B (+4.4% avg), Cosmos 3-8B, and Embodied-R1.5-8B. State-of-the-art on R2R-CE vision-and-language navigation (RGB-only setting) and strong zero-shot performance on Matterport3D Object Goal Navigation.

🧱 Model Card

| Field | Value | |---|---| | Architecture | HYV3VLForConditionalGeneration (VL wrapper over HYV3ForCausalLM MoE LLM + Hy-ViT2 vision encoder) | | Model type | hy_v3_vl | | Total parameters | ~30B | | Activated parameters per token | ~3B (8 of 128 experts + 1 shared) | | Context length | 32,768 tokens | | Precision | BF16 | | Vision inputs | Image (up to 128 per prompt); native aspect ratios | | Chat template | Unified chat_template.jinja bundled with weights (supports enable_thinking kwarg) |

🛠️ Dependencies and Installation

Prerequisites

• 🖥️ Operating System: Linux (recommended)
• 🐍 Python: 3.10+
• ⚡ CUDA: 12.x (H100 / H20 / A100 tested)
• 🔥 PyTorch: 2.4+
• 🎮 GPU: NVIDIA GPU(s). The full BF16 model requires ~86 GB across GPUs; a single 8×80 GB node is sufficient.

Installation

We pin dependencies to the versions we validated end-to-end (vllm==0.14.1 + transformers==4.57.6 + torch==2.9.1). The cleanest way to install these — with the CUDA build matched to your driver — is uv:

# Install uv once (skip if you already have it)
curl -LsSf https://astral.sh/uv/install.sh | sh

# Create a fresh venv (Python 3.10+)
uv venv --python 3.12
source .venv/bin/activate

# Clone the repo (provides the vLLM plugin and example scripts)
git clone https://github.com/Tencent-Hunyuan/HY-Embodied
cd HY-Embodied

🚀 Quick Start with vLLM (recommended)

vLLM is the recommended path for serving Hy-Embodied-VLM-1.0. Install vLLM together with matched torch/transformers wheels, then install this repo's plugin (registers the HYV3VL model and the reasoning / tool-call parsers):

# One-shot install: vllm + torch + torchvision + transformers at matching
# versions, with the CUDA...