ByteDance-Seed/par-protein

ByteDance-Seed/par-protein

Description: (ICML 2026 Oral) Protein Autoregressive Modeling via Multiscale Structure Generation, Official Implementation from ByteDance Seed

Language: Python

License: Apache-2.0

Stars: 1

Forks: 0

Open issues: 0

Created: 2026-06-16T07:36:23Z

Pushed: 2026-07-03T00:55:41Z

Default branch: main

Fork: no

Archived: no

README:

PAR: Protein Autoregressive Modeling via Multi-Scale Structure Generation

*Abstract.* We present protein autoregressive modeling (PAR), the first multi-scale autore- gressive framework for protein backbone generation via coarse-to-fine next-scale prediction. Using the hierarchical nature of proteins, PAR generates structures that mimic sculpting a statue, forming a coarse topology and refining structural details over scales. To achieve this, PAR consists of three key components: (i) multi- scale downsampling operations that represent protein structures across multiple scales during training; (ii) an autoregressive transformer that encodes multi-scale information and produces conditional embeddings to guide structure generation; (iii) a flow-based backbone decoder that generates backbone atoms conditioned on these embeddings. Moreover, autoregressive models suffer from exposure bias, caused by the training and the generation procedure mismatch, and substantially de- grades structure generation quality. We effectively alleviate this issue by adopting noisy context learning and scheduled sampling, enabling robust backbone gener- ation. Notably, PAR exhibits strong zero-shot generalization, supporting flexible human-prompted conditional generation and motif scaffolding without requiring fine-tuning. On the unconditional generation benchmark, PAR effectively learns protein distributions and produces backbones of high design quality, and exhibits favorable scaling behavior. Together, these properties establish PAR as a promising framework for protein structure generation.

Setup 🚀

You can use the pre-configured proteina environment, or build the environment using the following script:

# 1. make sure conda is available before running the following

# 2. change DATA_PATH and CKPT_HOME to your own path in prepare_env.sh
DATA_PATH=/your/own/data
CKPT_HOME=/your/own/ckpt/home

# 3. create conda env and activate it
source prepare_env.sh && conda activate PAR

# 4. install MPNN and ESMFold for inference
source script_utils/download_pmpnn_weghts.sh # Prepare PMPNN weights for designability metrics

For training and evaluation, additional meta files are required. Please refer to the proteina setup for more details.

Download Model Checkpoints 🤗

PAR model checkpoints are available on HuggingFace:

Download the checkpoints from ByteDance-Seed/PAR and set pretrain_ckpt_path to the downloaded checkpoint path for finetuning or inference.

Download D_FS And PDB Data 📦

For the D_FS dataset, you need to download the d_fs_index.txt following proteina setup. If you properly download the data files as in proteina setup, you will find the d_FS_index.txt under $DATA_PATH/D_FS/, which contains the indices of the D_FS dataset as created by Proteina team.

Check the data config configs/datasets_config/afdb/d_FS.yaml and make sure overwrite: False to avoid overwriting your processed data.

See [example_download_data.ipynb](example_download_data.ipynb) for more details.

Statistics: The D_FS dataset used for pretraining contains 0.6M structures, as processed by Proteina. This dataset ensures both high quality (pLDDT > 80) and structural diversity, with sequence lengths ranging from 32 to 256 residues. We follow Proteina and split it by 98:19:1 for training, validation and testing. The PDB dataset used for finetuning contains 21K designable (sc-RMSD <= 2) single chains after filtering. For detailed criterion, please refer to the Proteina paper.

Unconditional Pretraining 🧠

We use hydra to compose the training config. Taking configs/experiment_config/training_ca_par_60m.yaml as the example, we introduce the config structure as follows:

# training_ca_par_60m.yaml

...
# the following lines will load the model and data config
defaults:
- /model: ca_par_60m # caflow or frameflow
- _self_
- /datasets_config: afdb/d_FS

# by default, we use self conditioning, scheduled sampling, and noisy context learning for training
training:
self_cond: true
fold_cond: false
schedule_sampling: true
noisy_context_learning: true

The model config is located at configs/model/ca_par_60m.yaml

# ca_par_60m.yaml

...
# the following lines will load ca_af3_60M_notri_par as the transformer encoder and caflow as the flow decoder, which follow the same architecture as Proteina
# by default, we do not use aux loss and pair representation
# browse the configs/model/nn and configs/atom_diff/nn folders for more options
defaults:
- _self_
- nn: ca_af3_60M_notri_par
- /atom_diff: caflow_60m

# the default scale configuration
protein_decomposer:
structure_dim: 3 # Ca-only
num_local_centroids: [64, 128, 256]

You can simply reproduce our 60m pretraining using the following lines:

source prepare_env.sh && conda activate PAR
python proteinfoundation/train_par.py

This will train a 60m model on D_FS dataset for 200k steps with learning rate 0.0001.

We also provide a detailed training script run/run_training_par.sh for you to customize training.

# Enviroments handled inside
bash run/run_training_par.sh

PDB Finetuning 🔧

Change the following line in configs/experiment_config/finetune_ca_par_400m.yaml to specify the pretrained checkpoint.

pretrain_ckpt_path: /your/pretrained/model.ckpt

Then run the following lines to reproduce our PDB finetuning experiments:

source prepare_env.sh && conda activate PAR
python proteinfoundation/train_par.py experiment_config=finetune_ca_par_400m

By default, we only use the designable PDB structures for finetuning, as configured in configs/datasets_config/pdb/pdb_train.yaml:

filter_designable: True

This requires a pre-computed designable set of PDB codes, specified in...