arXiv:2509.09666cs.CV2025-09被引 10

用文本作中间表示,让图文理解生成任务相互提升。

Unified Multimodal Models as Auto-Encoders

  • 以文本为中间表示,统一图文理解与生成的自编码框架。
  • 通过重建奖励强化训练,使图文双向任务协同优化。
  • 提升小物体识别与复杂指令遵循能力,适合多模态研究者。

图像到文本(I2T)理解与文本到图像(T2I)生成是两个基础且重要的多模态任务,但传统上被独立处理,错失了相互增强的机会。本文提出,在共享自编码视角下,文本可作为连接两者的中间潜在表示:将图像编码为文本语义(I2T),再从文本解码回图像(T2I)。核心思想是:若编码器真正理解图像,则应保留全部关键结构;若解码器真正理解文本,则应忠实还原该结构。基于此,我们提出统一的后训练方法 Unified-GRPO,通过重构奖励进行强化学习联合优化,最大化输入与生成图像间的语义一致性。在此目标下,编码器被鼓励从图像中提取尽可能准确、全面的语义信息以提升重建质量,而解码器则根据编码器的先验进行生成,实现自我进化。实验表明,以文本为中间表示并采用重构式强化学习训练,显著提升了 I2T 的细粒度视觉感知能力(如小物体识别、定位等),其密集嵌入与语言先验也反过来增强了 T2I 的保真度与复杂指令遵循能力。结果证明,重构式强化学习在自编码框架内建立了跨模态的正向反馈机制。

原文摘要 · Abstract (English)

Image-to-text (I2T) understanding and text-to-image (T2I) generation are two fundamental, important yet traditionally isolated multimodal tasks. Despite their intrinsic connection, existing approaches typically optimize them independently, missing the opportunity for mutual enhancement. In this paper, we argue that the both tasks can be connected under a shared Auto-Encoder perspective, where text serves as the intermediate latent representation bridging the two directions - encoding images into textual semantics (I2T) and decoding text back into images (T2I). Our key insight is that if the encoder truly "understands" the image, it should capture all essential structure, and if the decoder truly "understands" the text, it should recover that structure faithfully. Building upon this principle, we propose Unified-GRPO, a post-training method based on reinforcement learning that jointly optimizes both modules through reconstructive rewards, maximizing the semantic consistency between the input and the generated images. Under this reconstruction objective, the encoder is encouraged to extract as much accurate and comprehensive semantic information from the input image to maximize reconstruction quality, while the decoder is simultaneously optimized to generate conditioned on the encoder's prior, enabling a self-evolving improvement. Empirically, we find that using text as the intermediate representation and training under a reconstructive RL paradigm effectively benefits both I2T and T2I. The I2T module gains stronger fine-grained visual perception, such as small-object recognition, grounding, etc, while its dense embeddings and language priors, in turn, provide richer semantic signals that improve T2I fidelity and complex instruction following. These results demonstrate that the reconstructive RL establishes a mutually reinforcing cross-modal synergy within the auto-encoding framework.

多模态自编码强化学习图文生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。