Tencent-Hunyuan/GEAR

Tencent-Hunyuan/GEAR

Language: Python

License: NOASSERTION

Stars: 20

Forks: 0

Open issues: 0

Created: 2026-06-30T08:45:16Z

Pushed: 2026-07-01T03:15:10Z

Default branch: main

Fork: no

Archived: no

README:

GEAR: Guided End-to-End AutoRegression for Image Synthesis

Official PyTorch Implementation

Bin Lin1,2,*, Zheyuan Liu1,2,*, Chenguo Lin1, Sixiang Chen2,*, Yunyang Ge1,2,*,

Yunlong Lin, Jianwei Zhang2, Miles Yang2, Zhao Zhong2, Liefeng Bo2, Li Yuan1,†

1 Peking University 2 Tencent Hunyuan

• Work done during internship at Tencent Hunyuan † Corresponding author

> 🚀 GEAR co-trains a VQ tokenizer and an autoregressive (AR) generator end-to-end — the tokenizer feels the gradient from its downstream generator, so the tokens it emits become much easier to predict. Up to ~10× faster ImageNet gFID convergence, drop-in across VQ / LFQ / IBQ and text-to-image.

✨ Highlights

• 🔗 Guided end-to-end. The soft-assignment bridge lets the AR *guide* the tokenizer, succeeding exactly where the straight-through estimator collapses; the next-token loss never touches the tokenizer.
• 🔄 Alignment flips to the AR. Opposite of diffusion-side REPA (REPA-E / VA-VAE): the tokenizer becomes *less* DINOv2-like and lower-entropy, while the AR's per-patch features track DINOv2 far more closely — reconstruction preserved.
• ⚡ Faster & better. ~10× faster ImageNet gFID convergence; on GPIC text-to-image a fresh AR on the frozen tokenizer hits the baseline's NTP loss 2.5× and REPA loss 11.1× faster, with better gFID at B / L / XL.
• 🧩 General & drop-in. Works across VQVAE / LFQ / IBQ and across class-conditional ImageNet and text-to-image — freeze the tuned tokenizer and drop it into a standard pipeline.

🔬 How it works

The blocker for end-to-end AR training is that the VQ index is a non-differentiable argmax (a straight-through estimator collapses, gFID≈105). GEAR fixes it with a dual read-out of the codebook: a hard one-hot branch trains the AR, while a differentiable soft branch softmax(-d/τ) carries a REPA loss back to update only the tokenizer. The two optimizations stay decoupled, with one coefficient λ and no crosstalk:

$$\theta_{\mathrm{tok}} \leftarrow \theta_{\mathrm{tok}} - \eta \nabla_{\theta_{\mathrm{tok}}} \left( \mathcal{L}_{\mathrm{VQ}} + \lambda \mathcal{L}^{\mathrm{s}}_{\mathrm{align}} \right)$$

$$\theta_{\mathrm{AR}} \leftarrow \theta_{\mathrm{AR}} - \eta \nabla_{\theta_{\mathrm{AR}}} \left( \mathcal{L}_{\mathrm{NTP}} + \lambda \mathcal{L}^{\mathrm{h}}_{\mathrm{align}} \right)$$

The tokenizer loss is the standard VQ objective (reconstruction + LPIPS + GAN + entropy + commit) plus the soft alignment term; the AR is trained by NTP plus the hard REPA term.

---

Table of contents

• [Installation](#installation)
• [Pretrained weights](#pretrained-weights)
• [Data preparation](#data-preparation)
• [Training](#training)
• [Inference](#inference)
• [Evaluation](#evaluation)
• [Citation](#citation)

---

Installation

conda create -n gear python=3.13 -y
conda activate gear

# 1) PyTorch first, matched to your CUDA (example: CUDA 12.8)
pip install torch==2.10.0 torchvision==0.25.0 torchaudio==2.10.0 \
--index-url https://download.pytorch.org/whl/cu128

# 2) everything else
pip install -r requirements.txt

The gear env above covers training, inference, and the in-repo GPIC FD-DINOv2 eval. The external benchmark scorers (GenEval, DPG-Bench, WISE judge, and the ADM ImageNet evaluator) each need their own conda environment — create them by following the per-benchmark READMEs linked under [Evaluation](#evaluation) before running those benchmarks.

---

Pretrained weights

GEAR's core contribution is the tokenizer, so that is what we release. Train your own AR on a frozen GEAR tokenizer with the [training scripts](#training).

For each public quantizer (VQ-16 / LFQ-16 / IBQ-16, all with a 16384-entry codebook) we release two tokenizers on the 🤗 Hub:

• Warm-up — a short reconstruction fine-tune that only recovers the GAN discriminator the

public weights omit. It is the baseline both LlamaGen-REPA and GEAR start from.

• GEAR — the same tokenizer after end-to-end fine-tuning with the AR.

| Quantizer | Warm-up (baseline) | GEAR (end-to-end) | |---|---|---| | VQ-16 | `Warmup-VQ` · vq-with-gan.pt | `GEAR-VQ` · gear-vq.pt | | LFQ-16 | `Warmup-LFQ` · lfq-with-gan.pt | `GEAR-LFQ` · gear-lfq.pt | | IBQ-16 | `Warmup-IBQ` · ibq-with-gan.pt | `GEAR-IBQ` · gear-ibq.pt |

huggingface-cli download BinLin203/GEAR-VQ --local-dir ckpts/GEAR-VQ

Reconstruction quality (ImageNet val)

The warm-up and end-to-end (GEAR) tokenizers both keep reconstruction performance on par with the original pretrained weights. *Original* is the public release.

| Quantizer | Setting | rFID↓ | PSNR↑ | SSIM↑ | |---|---|---|---|---| | VQ-16 | Original | 2.19 | 20.79 | 0.55 | | | Warm-up | 1.72 | 21.06 | 0.57 | | | GEAR | 1.64 | 20.78 | 0.56 | | LFQ-16 | Original | 2.82 | 21.47 | 0.58 | | | Warm-up | 2.42 | 20.97 | 0.56 | | | GEAR | 2.13 | 20.48 | 0.55 | | IBQ-16 | Original | 2.23 | 21.23 | 0.58 | | | Warm-up | 1.97 | 21.18 | 0.58 | | | GEAR | 1.72 | 20.92 | 0.57 |

> All rows use bicubic resize for an apples-to-apples comparison. The choice of > interpolation matters (the official LFQ / IBQ numbers use bilinear, which differs); see the > paper appendix for the full bilinear vs. bicubic table.

Original tokenizers (to reproduce the warm-up)

The *Original* rows above are the public pretrains; start from these to redo the warm-up step:

• VQ: `vq_ds16_c2i.pt` (LlamaGen)
• IBQ: `IBQ_pretrain_16384.ckpt` (TencentARC)
• LFQ: `pretrain256_16384.ckpt` (Open-MAGVIT2)

---

Data preparation

• ImageNet-1K (tokenizer + AR training): the...