Tencent-Hunyuan/Rosetta-inference

Tencent-Hunyuan/Rosetta-inference

Description: Open-source native multimodal pretraining — without catastrophic forgetting.

Language: Python

License: NOASSERTION

Stars: 0

Forks: 0

Open issues: 0

Created: 2026-06-26T03:31:42Z

Pushed: 2026-06-30T03:56:42Z

Default branch: main

Fork: no

Archived: no

README:

> Figure 1. *(Left)* Performance on MMLU (Language Ability) across composable pretraining stages (LM → +MMU → +T2I). Standard MoE and structurally-isolated MoT suffer catastrophic routing collapse upon integrating continuous generative objectives. Rosetta maintains a stable semantic anchor throughout all stages. *(Right)* Qualitative image generation results from the Rosetta model.

---

🏗️ Architecture

> Figure 2. Rosetta FFN. Three mechanisms enable non-destructive modality expansion: (1) Unified Attention — globally shared QKV projections preserve dense cross-modal interactions. (2) Composable FFN — modality-specific plug-and-play experts (Text / ViT / VAE) are bridged by a single Global Shared Expert that anchors foundational knowledge. (3) Conflict-Free Optimization (MAOP) — surgically neutralizes destructive gradients with zero memory overhead.

---

🚀 Quick Start

1. Environment Setup

Requirements: Python 3.12+, CUDA 12.8+

conda create -n rosetta python=3.12 -y
conda activate rosetta

# 1. Install PyTorch using your CUDA version
pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 --index-url https://download.pytorch.org/whl/cu128

# 2. Install dependencies
pip install -r requirements.txt

# 3. Install Flash Attention (required; takes ~20 mins to compile)
FLASH_ATTENTION_FORCE_BUILD=TRUE MAX_JOBS=8 pip install flash-attn==2.8.1 --no-build-isolation --no-binary flash-attn --no-cache-dir

2. Download Weights and Assets

Evaluation requires a model checkpoint and the shared assets (VAE, ViT, tokenizer, and evaluation datasets).

Step 1 — Download shared assets (required by all models, download once):

hf download tencent/Rosetta-inference public_assets.zip --local-dir . && unzip -o public_assets.zip && rm public_assets.zip

Step 2 — Download model checkpoint (example: main Rosetta model):

hf download tencent/Rosetta-inference --include "checkpoints/Rosetta-3.8B-A1B/**" --local-dir .

See [§ Model Checkpoints](#-model-checkpoints) for the full list of 15 checkpoints.

By default, the evaluation scripts look for assets under ./public_assets and checkpoints under ./checkpoints/Rosetta-3.8B-A1B. You can place them anywhere on disk and pass their locations via environment variables:

# Use the default ./public_assets and ./checkpoints/Rosetta-3.8B-A1B paths.
bash scripts/eval/eval_arc_c.sh

# Or override the paths explicitly.
ASSETS_BASE=/your/path/public_assets EXP=/your/path/checkpoints/Rosetta-3.8B-A1B bash scripts/eval/eval_arc_c.sh

ASSETS_BASE defaults to ./public_assets and EXP defaults to ./checkpoints/Rosetta-3.8B-A1B (relative to repo root) if not set.

Expected directory layout

/your/path/
├── checkpoints/
│ ├── Rosetta-3.8B-A1B/
│ │ └── hf_weights/ ← safetensors model weights
│ ├── MoE-3.8B-A1B/
│ └── MoT-4.5B-A1B/
└── public_assets/
├── image_encoder/ ← VAE (flux2-vae, ...)
├── vision_encoder/ ← ViT (Qwen3-VL-30B-A3B-Instruct)
├── pretrained_llm/ ← tokenizer + Qwen3-0.6B-Base
├── generation_configs/ ← generation settings for evaluation
└── evaluation/ ← benchmark datasets
├── ARC_C/
├── MMLU/
├── BBH/
├── MBPP/
├── MMMU/
├── MMBench/
├── POPE/
├── AI2D/
├── RealWorldQA/
└── T2I-CompBench/

---

📦 Model Checkpoints

The final 3 checkpoints are for benchmark reproduction:

| Checkpoint | Capabilities | Total / Active | HuggingFace | |:-----------|:-------------|:--------------:|:-----------:| | Rosetta-3.8B-A1B | LM + MMU + T2I | 3.8B / 0.97B | 🤗 Download | | MoE-3.8B-A1B | LM + MMU + T2I | 3.8B / 0.97B | 🤗 Download | | MoT-4.5B-A1B | LM + MMU + T2I | 4.5B / 0.97B | 🤗 Download |

Full checkpoint list — all 15 models across 3 architectures × 5 training stages, plus the MoT Stage 3 init checkpoint

| Checkpoint | Stage | Iter | Total / Active | HuggingFace | |:-----------|:------|-----:|:--------------:|:-----------:| | Rosetta-3.8B-A1B-init | Upcycling init | 0 | 3.8B / 0.97B | 🤗 Download | | Rosetta-3.8B-A1B-stage1-lm | LM | 35K | 3.8B / 0.97B | 🤗 Download | | Rosetta-3.8B-A1B-stage2-lm-mmu-warmup | LM+MMU warmup | 3K | 3.8B / 0.97B | 🤗 Download | | Rosetta-3.8B-A1B-stage2-lm-mmu | LM+MMU | 20K | 3.8B / 0.97B | 🤗 Download | | Rosetta-3.8B-A1B | LM+MMU+T2I | 400K | 3.8B / 0.97B | 🤗 Download | | MoE-3.8B-A1B-init | Upcycling init | 0 | 3.8B / 0.97B | 🤗 Download | | MoE-3.8B-A1B-stage1-lm | LM | 35K | 3.8B / 0.97B | 🤗 Download | | MoE-3.8B-A1B-stage2-lm-mmu-warmup | LM+MMU warmup | 3K | 3.8B / 0.97B | 🤗 Download | | MoE-3.8B-A1B-stage2-lm-mmu | LM+MMU | 20K | 3.8B / 0.97B | 🤗 Download | | MoE-3.8B-A1B | LM+MMU+T2I | 400K | 3.8B / 0.97B | 🤗 Download | | MoT-4.5B-A1B-init | Upcycling init | 0 | 4.5B / 0.97B | 🤗 Download | | MoT-4.5B-A1B-stage1-lm | LM | 35K | 4.5B / 0.97B |...