Tencent-Hunyuan/Prism

Tencent-Hunyuan/Prism

Description: We propose Prism, a dynamic sparse attention framework for natively training joint video-audio generation models at 2K.

Language: Python

License: NOASSERTION

Stars: 14

Forks: 1

Open issues: 0

Created: 2026-09-21T11:31:58Z

Pushed: 2026-10-06T01:54:28Z

Default branch: main

Fork: no

Archived: no

README:

Prism

Prism:Dynamic Sparse Attention for Native 2K Joint Video-Audio Generation Model Training

Shuyuan Tu1, Qi Tian2, Yinming Huang1, Yue Wu2, Xintong Han2, Kaihang Pan3, Weijie Kong2, Jiangfeng Xiong2, Jian-Wei Zhang2, Zuxuan Wu1, Yu-Gang Jiang1

[1Fudan University; 2Tencent Hunyuan; 3Zhejiang University]

Overview

The overview of the framework of Prism.

Native training joint video-audio generation models at higher resolutions empowers them to learn richer visual details and sharper motion dynamics. However, full attention incurs quadratic cost and, as resolution increases, spreads attention over increasingly redundant tokens, diluting learning signals for informative content and disrupting pretrained priors. Existing sparse attention methods either target training-free acceleration or overlook the unique structure of joint video-audio data, where cross-modal interactions are inherently concentrated around sound-producing regions. To address this, we propose Prism, a dynamic sparse attention framework for natively training joint video-audio generation models at 2K. In particular, Prism organizes the token sequence into spatiotemporal macro-zones, enabling the attention structure to adapt to local content. For each zone, it estimates local information structure via video feature variance along the channel and feature norms from the audio-to-video cross-attention, jointly capturing how visual content varies directionally and how strongly audio influences each visual region. Based on these signals, Prism dynamically assigns a tailored block shape to each zone, applying finer partitioning along axes of rapid visual content variation and strong audio-visual coupling. This encourages tokens within each block to remain semantically coherent, allowing block-level features to capture both visual content and joint video-audio interaction patterns. Prism further adopts a hybrid block selection strategy to dynamically determine per-query sparsity. Experiments show that Prism achieves 2.5$\times$ training speedup compared to full attention, while surpassing it in generation quality.

News

• [2026-x-xx]:🔥 The project page, training/inference code, technical report and a preview model checkpoint are released. Stay tuned!

🛠️ To-Do List

• [x] Prism-preview-alpha (stable)
• [x] Prism-preview-beta (motion)
• [x] Data Pre-Processing Code (Latent Extraction and Latent Decode)
• [x] Training Code
• [x] Full Finetuning Code
• [x] Inference Code
• [ ] Prism-pro

🔑 Quickstart

The Preview Model supports native joint video-audio generation at 720p, 1080p, and 2K resolutions, as well as native joint video-audio training at these resolutions.

🧱 Environment Setup

Prerequisites. Prism requires Python ≥ 3.10, CUDA ≥ 12.4, and a system-level ffmpeg installation.

# 1 — Install PyTorch (CUDA 12.4)
pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 --index-url https://download.pytorch.org/whl/cu124
# 2 — Install core dependencies
pip install -r requirements.txt
# 3 — Install FlashAttention (requires CUDA)
pip install flash-attn --no-build-isolation
# 4 — Install `ffmpeg` (system-level)
sudo apt-get install ffmpeg
# 5 — Verify the installation
python -c "
import torch, diffusers, transformers, flash_attn, triton
from audiotools import AudioSignal
from decord import VideoReader
print(f'torch {torch.__version__}')
print(f'diffusers {diffusers.__version__}')
print(f'transformers {transformers.__version__}')
print(f'flash_attn {flash_attn.__version__}')
print(f'triton {triton.__version__}')
print('All imports OK ✅')"

(Multi-node only) Quick cluster-wide install via `pdsh`

If you are running on a multi-node cluster, the following one-liner installs the key packages across all nodes in parallel:

export node_ip=$(echo ${NODE_IP_LIST} | sed 's/:8//g')
pdsh -f 256 -w $node_ip "pip install ipdb yunchang mmengine"
pdsh -f 256 -w $node_ip "pip uninstall diffusers -y && pip install diffusers==0.33.0"
pdsh -f 256 -w $node_ip "pip install descript-audiotools"
pdsh -f 256 -w $node_ip "pip install imageio[ffmpeg] imageio[pyav]"
pdsh -f 256 -w $node_ip "pip install decord"
pdsh -f 256 -w $node_ip "pip install flash-attn --no-build-isolation"

🧱 Download Weights

If you encounter connection issues with Hugging Face, you can utilize the mirror endpoint by setting the environment variable: export HF_ENDPOINT=https://hf-mirror.com. Please download weights manually as follows:

pip install "huggingface_hub[cli]"
cd Prism
mkdir checkpoints
huggingface-cli download FrancisRing/Prism --local-dir ./checkpoints

All the weights should be organized as follows. The overall file structure of this project should be organized as follows:

Prism/
├── configs/
├── hymm/
├── scripts/
├── tools/
├── checkpoints/
│ ├── pretrained_models/
│ │ └── MOVA-360p/ # base MOVA pretrained model
│ ├── preview_alpha/ # Prism-preview-alpha weights
│ │ └── diffusion_pytorch_model.safetensors
│ └── preview_beta/ # Prism-preview-beta weights
│ └── diffusion_pytorch_model.safetensors
├── mova_infer.sh
├── mova_infer_fsdp.sh
├── requirements.txt
└── README.md

🧱 Training Dataset Preparation

Training reads a plain CSV file (one row per clip). The row details are depicted as follows:

| column | required | meaning | | --- | --- | --- | | latent_path | yes | cached WAN VAE latent .npy, shape [C, T, H, W] | |...