OpenBMB/MiniCPM-Robot

OpenBMB/MiniCPM-Robot

Description: A Smarter and Faster On-Device AI Brain for Robots

Language: Python

License: Apache-2.0

Stars: 6

Forks: 0

Open issues: 0

Created: 2026-07-18T19:35:23Z

Pushed: 2026-07-18T19:37:34Z

Default branch: main

Fork: no

Archived: no

README:

A Smarter and Faster On-Device AI Brain for Robots

English · 中文

X |

Discord |

MiniCPM-RobotManip🤗 | MiniCPM-RobotTrack🤗 | 🍳 Cookbook

MiniCPM-Robot is MiniCPM's embodied intelligence model family for perception, decision-making, and action in the physical world, advancing MiniCPM from multimodal understanding toward real-world interaction. The first models in the family include:

• MiniCPM-RobotManip: 🦾 Designed for generalist robot manipulation in simulation and the real world. Built on MiniCPM-V 4.6 as a 1.5B-parameter general-purpose embodied model, it uses one set of weights across all downstream tasks and outperforms larger models such as π₀.₅ and Qwen-VLA across representative evaluations. It inherits efficient visual encoding and visual token compression, while streaming inference continuously incorporates historical observations into the model context, reducing per-step compute from 125 TFLOPs to 3.3 TFLOPs while retaining 60 frames of history and supporting up to one minute of visual memory. This moves VLA beyond reactive action generation from single-frame observations toward continuous decision-making grounded in long-horizon visual context.

• MiniCPM-RobotTrack: 🎯 The first fully on-device embodied target-tracking solution, covering static-target, dynamic-target, and adversarial-target settings. Built on MiniCPM4-0.5B with 0.9B total parameters, it improves robustness in long-tail scenarios through a self-evolving data pipeline and DAgger-style closed-loop training, achieving state-of-the-art performance among open-source models on EVT-Bench. End-to-end system optimization enables 5+ FPS with approximately 180 ms latency on the Unitree Go2 EDU's onboard compute, delivering fully local, vision-only natural-language tracking.

📰 News

• [2026.07.19] 🔥🔥🔥 We release and open-source MiniCPM-Robot, MiniCPM's first embodied intelligence model family for interaction with the physical world. Its first releases are MiniCPM-RobotManip for generalist robot manipulation and MiniCPM-RobotTrack for embodied target tracking. Try it now!

• [2026.07.19] 🚀🚀🚀 PhyAI adds Day-0 support for MiniCPM-Robot, increasing inference throughput on NVIDIA H20 from 10.12 Hz to 36.77 Hz through CUDA Graph and custom Triton fused kernels.

Contents

• [MiniCPM-RobotManip](#minicpm-robotmanip)
• [Benchmark Results](#benchmark-results)
• [Quick Start](#quick-start)
• [Inference](#inference)
• [MiniCPM-RobotTrack](#minicpm-robottrack)
• [Examples](#examples)
• [EVT-Bench Results](#evt-bench-results)
• [Quick Start](#quick-start-1)
• [Data Preparation](#data-preparation)
• [Finetuning](#finetuning)
• [Evaluation](#evaluation)
• [Go2 Deployment](#go2-deployment)
• [Model Zoo](#model-zoo)
• [License](#license)
• [Acknowledgments](#acknowledgments)

MiniCPM-RobotManip

MiniCPM-RobotManip is a 1.5B vision-language-action model for embodied manipulation with the following highlights:

Generalist Manipulation: A unified 1.5B generalist policy that covers all downstream tasks with one set of weights.

Streaming Context: Historical observations are incorporated into the model context through streaming inference. With 60 frames of history, traditional recomputation requires 125 TFLOPs per decision step, while streaming inference needs only 3.3 TFLOPs. The model supports up to 1 minute of visual context memory while keeping online cost comparable to traditional single-frame reactive inference.

Efficient Inference: Inherits MiniCPM-V 4.6's visual token compression, reducing each frame from 256 to 64 visual tokens for 4× compression. With H100, BF16, and single-frame input, model-forward latency per decision step is 120 ms, compared with 234 ms for π0.5. The measurement excludes task autoregressive decoding.

Benchmark Results

Method Eval Setting Open Weights Model Size LIBERO Calvin (ABC→D) RoboTwin2 (clean+random) RMBench

easyhard

π₀Specialist✅3B94.43.965.958.4—

π₀.₅Specialist✅3B96.94.182.776.810.4

Abot-M0Specialist✅4B+98.6—86.185.1—

StarVLA-αGeneralist✅4B+97.8—88.787.8—

Qwen-VLAGeneralist❌5B+97.9—86.187.2—

LingBot-VASpecialist✅5B+98.5—92.991.6—

MiniCPM-RobotManipGeneralist✅1.5B97.54.191.391.653.3

Quick Start

Install and initialize Conda first. The specification follows the tested Python 3.10 and PyTorch 2.6.0 (CUDA 12.4) setup.

cd MiniCPM-RobotManip conda env create -f environment.yml conda activate MiniCPM-RobotManip

Inference

Run single-sample inference with vla_infer.py. Provide at least one image and a language instruction; robot state defaults to zeros if omitted. The model returns an action chunk of shape (30, 80).

cd MiniCPM-RobotManip python vla_infer.py \ --image frame.jpg \ --text "Pick up the red block." \ --checkpoint ./checkpoint \ --state-file state.npy \ --embodiment-id 0 \ --output action.npy

Use multiple --image flags for multi-view inputs. Without --output, the predicted action is printed as JSON.

cd MiniCPM-RobotManip python vla_infer.py \ --image cam_front.jpg \ --image cam_wrist.jpg \ --text "Pick up the red block." \ --checkpoint ./checkpoint

MiniCPM-RobotTrack

MiniCPM-RobotTrack is a compact vision-language-action policy for embodied target tracking built on MiniCPM4-0.5B with following highlights:

Quality-driven self-evolving data pipeline: automated checks and manual review remove abnormal trajectories, incorrect actions, and invalid interactions, while continual model-environment interaction adds high-value training samples.

DAgger for embodied tracking: after learning from large-scale general scenarios, the model interacts with simulators and real robots to expose failures in long-tail cases such as target crossings, rapid turns, short occlusions, and multi-person intersections. Samples corrected by expert policies or rules are aggregated into the next training round to continuously improve tracking and generalization.

End-to-end Go2 optimization: joint optimization across visual...