arXiv:2509.07295cs.CVcs.AI2025-09中稿 · ICLR被引 48

用视觉编码器嵌入做密集提示,提升多模态模型生成与理解一致性。

Reconstruction Alignment Improves Unified Multimodal Models

  • 用自身视觉编码嵌入作为密集文本提示,自监督重建图像实现对齐
  • 仅27 GPU小时即显著提升生成与编辑性能(如GenEval从0.73→0.90)
  • 适用于各类多模态模型,适合追求高效微调的开发者

统一多模态模型(UMMs)将视觉理解与生成统一于单一架构中。然而传统训练依赖图像-文本对,其描述常稀疏且遗漏细粒度视觉信息,即便使用数百词也难以充分描述简单图像。本文提出重建对齐(RECA),一种资源高效的后训练方法,利用视觉理解编码器的嵌入作为密集“文本提示”,在无标注情况下提供丰富监督。具体地,RECA以模型自身的视觉嵌入为条件,优化其自监督重建输入图像,从而重新对齐理解与生成能力。尽管方法简单,但适用性广泛:在自回归、掩码自回归及扩散类多模态模型上均一致提升生成与编辑保真度。仅需27 GPU小时,后训练即显著提升生成性能(GenEval从0.73升至0.90,DPGBench从80.93升至88.15),同时改善编辑指标(ImgEdit从3.38升至3.75,GEdit从6.94升至7.27)。值得注意的是,RECA超越了更大规模的开源模型,适用于多种多模态架构,成为一种高效通用的后训练对齐策略。

原文摘要 · Abstract (English)

Unified multimodal models (UMMs) unify visual understanding and generation within a single architecture. However, conventional training relies on image-text pairs (or sequences) whose captions are typically sparse and miss fine-grained visual details, even when they use hundreds of words to describe a simple image. We introduce Reconstruction Alignment (RECA), a resource-efficient post-training method that leverages visual understanding encoder embeddings as dense "text prompts", providing rich supervision without captions. Concretely, RECA conditions a UMM on its own visual understanding embeddings and optimizes it to reconstruct the input image with a self-supervised reconstruction loss, thereby realigning understanding and generation. Despite its simplicity, RECA is broadly applicable: across autoregressive, masked-autoregressive, and diffusion-based UMMs, it consistently improves generation and editing fidelity. With only 27 GPU hours, post-training with RECA substantially improves image generation performance on GenEval (0.73 $\rightarrow$ 0.90) and DPGBench (80.93 $\rightarrow$ 88.15), while also boosting editing benchmarks (ImgEdit 3.38 $\rightarrow$ 3.75, GEdit 6.94 $\rightarrow$ 7.27). Notably, RECA surpasses much larger open-source models and applies broadly across diverse UMM architectures, establishing it as an efficient and general post-training alignment strategy for UMMs.

多模态图像生成自监督对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。