给自回归图像生成模型注入未来信息,显著加速收敛并提升质量
Mirai: Autoregressive Visual Generation Needs Foresight
- 通过引入未来位置的显式或隐式信息,改进模型因果建模
- 在ImageNet上使LlamaGen-B收敛速度提升10倍,FID从5.34降至4.34
- 无需修改结构,无推理开销,适合追求高效生成的研究者
自回归视觉生成器将图像建模为离散标记序列,采用仅依赖下一标记的因果监督。这种严格因果训练使每一步仅基于下一个标记优化,削弱全局一致性并减缓收敛。我们探究了引入未来信息(即非因果信号)是否能改善自回归视觉生成。通过控制实验,考察了信息注入程度、布局与来源,发现:当未来信息与模型在2D图像网格上的内部表征对齐时,可提升因果建模能力。为此提出Mirai(日语意为‘未来’)框架,无需改变架构且无额外推理开销:Mirai-E使用单向表示中多个未来位置的显式未来信息;Mirai-I则利用匹配双向表示中的隐式未来信息。大量实验表明,Mirai显著加速收敛并提升生成质量。例如,在ImageNet类条件图像生成任务中,其可使LlamaGen-B收敛速度提升最高达10倍,生成FID由5.34降至4.34。研究强调视觉自回归模型需要预见性。
原文摘要 · Abstract (English)
Autoregressive (AR) visual generators model images as sequences of discrete tokens and are trained with a next-token likelihood objective. This strict causal supervision optimizes each step based only on the immediate next token, which can weaken global coherence and slow convergence. We investigate whether foresight, training signals that originate from later tokens, can improve autoregressive visual generation. We conduct a series of controlled diagnostics along the injection level, foresight layout, and foresight source axes, revealing a key insight: aligning foresight with AR models' internal representations on the 2D image grid improves causal modeling. We formulate this insight with Mirai (meaning "future" in Japanese), a general framework that injects future information into AR training with no architecture change and no extra inference overhead: Mirai-E uses explicit foresight from multiple future positions of unidirectional representations, whereas Mirai-I leverages implicit foresight from matched bidirectional representations. Extensive experiments show that Mirai significantly accelerates convergence and improves generation quality. For instance, Mirai can speed up LlamaGen-B's convergence by up to 10$\times$ and reduce the generation FID from 5.34 to 4.34 on the ImageNet class-condition image generation benchmark. Our study highlights that visual autoregressive models need foresight.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。