arXiv:2606.11188cs.CV2026-06被引 1

统一视觉语言生成与编辑的自回归大模型,用离散表示实现跨任务协同。

ARM: An AutoRegressive Large Multimodal Model with Unified Discrete Representations

论文配图:ARM: An AutoRegressive Large Multimodal Model with Unified Discrete Representations
图 1 · 摘自论文原文
  • 用多目标监督训练图像离散分词器,构建共享语义空间。
  • 70亿参数模型在文本到图像生成上提升显著(如WISE从0.50升至0.56)。
  • 强化学习优化指令遵循与编辑一致性,促进生成与编辑任务协同提升。

本文提出ARM,一种基于离散表示的自回归多模态模型,统一图像理解、生成与编辑任务。首先,训练一个具有多目标监督的离散语义视觉分词器,将图像映射为紧凑的令牌序列,联合提升语义可区分性、语言对齐性和重建保真度,支持多样任务共享潜在空间。在此基础上,使用大规模文本与图像令牌序列训练一个70亿参数的自回归模型,无缝发展视觉-语言感知与生成能力。最后,为提升文本到图像生成和指令引导编辑的偏好对齐行为,采用强化学习优化任务级目标,如视觉质量、指令遵循度与编辑一致性。结果表明,强化学习不仅显著提升目标任务表现(如WISE整体得分从0.50提升至0.56,GEdit-Bench-EN G_O从5.75升至6.68),还引发生成与编辑间的跨任务协同效应。这些发现表明,结合强表示与偏好优化的自回归建模,是多模态智能可扩展的基础。

原文摘要 · Abstract (English)

This paper introduces ARM, a discrete representation-based AutoRegressive Model that unifies image understanding, generation, and editing within a next-token prediction framework. ARM is built on three efforts: first, we train a discrete semantic visual tokenizer that maps images into compact token sequences. Our tokenizer is supervised with multiple objectives that jointly promote semantic discriminability, language alignment and faithful reconstruction, thereby supporting diverse tasks in a shared latent space. With this, we train a 7B autoregressive model over large-scale text and image token sequences, seamlessly developing vision-language perception and generation capabilities. Finally, to further improve preference-aligned behavior for text-to-image generation and instruction-guided editing, ARM applies reinforcement learning (RL) to optimize task-level objectives such as visual quality, instruction adherence, and edit consistency. Surprisingly, the results show that RL not only substantially improves performance on the target tasks (e.g., raising WISE overall from 0.50 to 0.56, GEdit-Bench-EN G_O from 5.75 to 6.68), but also induces cross-task synergy between text-to-image generation and editing. Collectively, these findings highlight autoregressive modeling, when paired with strong representations and preference optimization, as a scalable foundation for multimodal intelligence. Code: https://github.com/wdrink/ARM.

多模态自回归生成模型强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。