让图像生成的分词器和生成器一起训练,更快更准。
GEAR: Guided End-to-End AutoRegression for Image Synthesis

- 分词器与生成器端到端联合训练,用对齐损失引导优化。
- 相比基线模型,ImageNet生成质量提升10倍收敛速度。
- 适用于多种分词器,适合图像生成与图文生成任务。
视觉生成模型通常分两阶段训练:先训练分词器用于重建并冻结,再在离散索引或连续隐变量上训练生成器。这种分离使分词器不了解生成器建模的难易程度。我们提出 GEAR(Guided End-to-End AutoRegression),实现向量量化(VQ)分词器与自回归(AR)生成器的联合端到端训练,并通过表示对齐进行引导。主要挑战在于,输入生成器的 VQ 索引不可导,梯度无法回传至分词器,传统直通估计器会失效。GEAR 通过代码本分配的双重读出解决此问题:硬分支(one-hot)用于生成器的下一步预测训练;软分支携带表示对齐损失,反向传播仅指导分词器。生成器因此引导分词器向其更易预测的索引分布演进。这将对齐负担从分词器转移到生成器:分词器特征不再像 DINOv2,而生成器特征变得更具语义性,与扩散模型方案相反。GEAR 相比强基线 LlamaGen-REPA,ImageNet gFID 收敛速度提升最高达 10 倍,学习到更优的局部与空间一致性特征,并可泛化至 VQVAE、LFQ、IBQ 等多种分词器及文本到图像生成任务。
原文摘要 · Abstract (English)
Visual generative models are typically trained in two stages. A tokenizer is first trained for reconstruction and then frozen, after which a generator is trained on its discrete indices or continuous latents. This decoupling leaves the tokenizer unaware of what the generator finds easy to model. We present GEAR (Guided End-to-end AutoRegression), which trains a vector-quantized (VQ) tokenizer and an autoregressive (AR) generator jointly and end-to-end, guided by representation alignment. The key obstacle is that the VQ index fed to the AR model is non-differentiable, so gradients cannot reach the tokenizer, and a straight-through estimator collapses. GEAR resolves this with a dual read-out of the codebook assignment. A hard, one-hot branch trains the AR with next-token prediction, while a differentiable soft branch carries a representation-alignment loss that flows back to guide only the tokenizer. The AR model thereby steers its tokenizer toward an index distribution it can predict more easily. This shifts the alignment burden from the tokenizer to the AR: the tokenizer's own features become less DINOv2-like while the AR's become more so, the opposite of diffusion-side recipes that make the latent itself semantic. GEAR speeds up ImageNet gFID convergence by up to 10x relative to the strong LlamaGen-REPA baseline, learns markedly better patch-level and spatially-coherent features, and generalizes across quantizers (VQVAE, LFQ, IBQ) and to text-to-image generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。