通过一致性正则化提升视觉自回归模型生成质量
REAR: Rethinking Visual Autoregressive Models via Generator-Tokenizer Consistency Regularization
- 引入逐标记正则化，让生成器与分词器对齐
- ImageNet上gFID降至1.86，IS提升至316.9
- 小模型（177M）媲美大扩散模型，无需改动架构
视觉自回归生成为统一视觉与语言模型提供了有前景的路径，但其性能仍逊于扩散模型。以往工作常将差距归因于分词器限制和栅格化顺序。本文从生成器-分词器不一致的角度识别出核心瓶颈：自回归生成的标记可能无法被分词器有效解码。为此，提出reAR，一种无需修改分词器、生成顺序、推理流程或外部模型的简单训练策略。该方法在预测下一个标记时，同时训练因果变换器恢复当前标记的视觉嵌入，并在噪声上下文中预测目标标记的嵌入。尽管结构简单，reAR显著提升性能：在ImageNet上，使用标准栅格化分词器时，gFID从3.02降至1.86，IS提升至316.9；应用于先进分词器时，仅用177M参数即达到1.42的gFID，媲美675M参数的前沿扩散模型。
原文摘要 · Abstract (English)
Visual autoregressive (AR) generation offers a promising path toward unifying vision and language models, yet its performance remains suboptimal against diffusion models. Prior work often attributes this gap to tokenizer limitations and rasterization ordering. In this work, we identify a core bottleneck from the perspective of generator-tokenizer inconsistency, i.e., the AR-generated tokens may not be well-decoded by the tokenizer. To address this, we propose reAR, a simple training strategy introducing a token-wise regularization objective: when predicting the next token, the causal transformer is also trained to recover the visual embedding of the current token and predict the embedding of the target token under a noisy context. It requires no changes to the tokenizer, generation order, inference pipeline, or external models. Despite its simplicity, reAR substantially improves performance. On ImageNet, it reduces gFID from 3.02 to 1.86 and improves IS to 316.9 using a standard rasterization-based tokenizer. When applied to advanced tokenizers, it achieves a gFID of 1.42 with only 177M parameters, matching the performance with larger state-of-the-art diffusion models (675M).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。