无需改架构，让大模型生成更连贯的图文内容。
Unleashing the Potential of Large Language Models for Text-to-Image Generation through Autoregressive Representation Alignment
- 用全局对齐损失和混合令牌<HYBNEXT>，让语言模型隐式学习空间与上下文一致性。
- 训练新模型时FID降低16.6%（ImageNet），从纯文本模型训练时降25.5%（MIMIC-CXR）。
- 适合想提升图文生成质量但不想动模型结构的研究者和开发者。
我们提出自回归表征对齐（ARRA），一种无需修改架构即可实现全局一致图文生成的新训练框架。与需复杂结构调整的以往方法不同，ARRA通过全局视觉对齐损失和混合令牌<HYBNEXT>，将语言模型的隐藏状态与外部视觉基础模型的表征对齐。<HYBNEXT>同时施加局部词预测与全局语义蒸馏双重约束，使语言模型在保持原有自回归范式的同时，隐式学习空间与上下文一致性。大量实验验证了ARRA的即插即用特性：从头训练时，对LlamaGen等自回归语言模型，图像生成在ImageNet上FID降低16.6%，LAION-COCO上降低12.0%；从仅文本生成的语言模型训练时，对Chameleon等先进模型，MIMIC-CXR上FID降低25.5%，DeepEyeNet上降低8.8%；用于领域适配时，将通用语言模型与BioMedCLIP等专用模型对齐，在医疗影像数据集MIMIC-CXR上相比直接微调降低18.6% FID。结果表明，训练目标重构比架构修改更能解决跨模态全局一致性挑战。该方法为推进自回归模型提供了一种互补范式。代码已开源：https://github.com/HKU-HealthAI/ARRA。
原文摘要 · Abstract (English)
We present Autoregressive Representation Alignment (ARRA), a new training framework that unlocks global-coherent text-to-image generation in autoregressive LLMs without architectural modifications. Different from prior works that require complex architectural redesigns, ARRA aligns LLM's hidden states with visual representations from external visual foundational models via a global visual alignment loss and a hybrid token, <HYBNEXT>. This token enforces dual constraints: local next-token prediction and global semantic distillation, enabling LLMs to implicitly learn spatial and contextual coherence while retaining their original autoregressive paradigm. Extensive experiments validate ARRA's plug-and-play versatility. When training T2I LLMs from scratch, ARRA reduces FID by 16.6% (ImageNet), 12.0% (LAION-COCO) for autoregressive LLMs like LlamaGen, without modifying original architecture and inference mechanism. For training from text-generation-only LLMs, ARRA reduces FID by 25.5% (MIMIC-CXR), 8.8% (DeepEyeNet) for advanced LLMs like Chameleon. For domain adaptation, ARRA aligns general-purpose LLMs with specialized models (e.g., BioMedCLIP), achieving an 18.6% FID reduction over direct fine-tuning on medical imaging (MIMIC-CXR). These results demonstrate that training objective redesign, rather than architectural modifications, can resolve cross-modal global coherence challenges. ARRA offers a complementary paradigm for advancing autoregressive models. The code is available at https://github.com/HKU-HealthAI/ARRA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。