让图文模型理解与生成能力更均衡，显著缩小两者差距。
HermesFlow: Seamlessly Closing the Gap in Multimodal Understanding and Generation
- 用同源数据构建理解与生成的偏好对齐数据
- 通过成对DPO与自对弈优化，实现双向能力同步提升
- 适用于下一代多模态大模型的通用对齐框架
自回归范式在多模态大语言模型（MLLM）中取得显著进展，如Show-o、Transfusion和Emu3等模型在统一图像理解和生成任务上表现优异。我们首次发现：多数MLLM的理解能力普遍强于生成能力，两者间存在显著差距。基于此洞察，我们提出HermesFlow——一种简单且通用的框架，旨在无缝弥合理解与生成之间的鸿沟。具体而言，以同源数据为输入，构建理解与生成的同源偏好数据；通过成对DPO与自对弈迭代优化，利用同源偏好数据有效对齐多模态理解与生成能力。大量实验证明，该方法显著优于先前方法，尤其在缩小多模态理解与生成差距方面表现突出。这些结果表明HermesFlow可作为下一代多模态基础模型的通用对齐框架。代码已开源：https://github.com/Gen-Verse/HermesFlow。
原文摘要 · Abstract (English)
The remarkable success of the autoregressive paradigm has made significant advancement in Multimodal Large Language Models (MLLMs), with powerful models like Show-o, Transfusion and Emu3 achieving notable progress in unified image understanding and generation. For the first time, we uncover a common phenomenon: the understanding capabilities of MLLMs are typically stronger than their generative capabilities, with a significant gap between the two. Building on this insight, we propose HermesFlow, a simple yet general framework designed to seamlessly bridge the gap between understanding and generation in MLLMs. Specifically, we take the homologous data as input to curate homologous preference data of both understanding and generation. Through Pair-DPO and self-play iterative optimization, HermesFlow effectively aligns multimodal understanding and generation using homologous preference data. Extensive experiments demonstrate the significant superiority of our approach over prior methods, particularly in narrowing the gap between multimodal understanding and generation. These findings highlight the potential of HermesFlow as a general alignment framework for next-generation multimodal foundation models. Code: https://github.com/Gen-Verse/HermesFlow
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。