用表征自编码器提升自回归模型生成效果,突破传统范式限制。
RAE-AR: Taming Autoregressive Models with Representation Autoencoders
- 通过分布归一化简化令牌,降低自回归建模难度。
- 训练时注入高斯噪声缓解暴露偏差,提升预测鲁棒性。
- 首次实现表征自编码器在自回归模型中媲美传统VAE的效果。
生成模型的潜在空间长期由变分自编码器(VAE)编码器主导。先前认为预训练表征编码器(如DINO、SigLIP、MAE)的潜在表示不适用于生成建模。最近,RAE方法带来希望,表明表征自编码器也能达到与VAE相当的性能。然而,将表征自编码器融入连续自回归(AR)模型的研究仍不充分。本文研究了在AR框架下使用高维表征自编码器的挑战,提出RAE-AR。针对自回归模型的特性,识别出两大障碍:复杂的逐令牌分布建模和因高维度放大的训练-推理差距(暴露偏差)。为此,我们引入分布归一化的令牌简化以降低建模难度并促进收敛;同时,在训练中加入高斯噪声注入,以增强预测鲁棒性、缓解暴露偏差。实验结果表明,这些改进显著缩小了性能差距,使表征自编码器在自回归模型上的表现可与传统VAE相媲美。本工作为视觉理解与生成建模的统一架构开辟了新路径。
原文摘要 · Abstract (English)
The latent space of generative modeling is long dominated by the VAE encoder. The latents from the pretrained representation encoders (e.g., DINO, SigLIP, MAE) are previously considered inappropriate for generative modeling. Recently, RAE method lights the hope and reveals that the representation autoencoder can also achieve competitive performance as the VAE encoder. However, the integration of representation autoencoder into continuous autoregressive (AR) models, remains largely unexplored. In this work, we investigate the challenges of employing high-dimensional representation autoencoders within the AR paradigm, denoted as \textit{RAE-AR}. We focus on the unique properties of AR models and identify two primary hurdles: complex token-wise distribution modeling and the high-dimensionality amplified training-inference gap (exposure bias). To address these, we introduce token simplification via distribution normalization to ease modeling difficulty and improve convergence. Furthermore, we enhance prediction robustness by incorporating Gaussian noise injection during training to mitigate exposure bias. Our empirical results demonstrate that these modifications substantially bridge the performance gap, enabling representation autoencoder to achieve results comparable to traditional VAEs on AR models. This work paves the way for a more unified architecture across visual understanding and generative modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。