用自监督双奖励机制,让多模态模型理解与生成能力互相提升。
SUDER: Self-Improving Unified Large Multimodal Models for Understanding and Generation with Dual Self-Rewards
- 利用理解与生成的互逆关系,通过模型内部反馈自我优化。
- 在文本转图像任务中性能显著提升,无需外部标注或人工反馈。
- 适合追求零样本优化、自进化多模态系统的研究者与开发者。
基于大语言模型,近期的大规模多模态模型(LMMs)将跨模态理解和生成统一于单一框架。然而,现有LMMs仍难以实现精准的视觉-语言对齐,常出现生成内容与视觉输入矛盾或未能遵循文本到图像提示的问题。当前解决方案依赖外部监督(如人工反馈或奖励模型),且仅适用于单向任务——理解或生成。本文基于理解与生成为天然互逆任务的观察,提出SUDER(Self-improving Unified LMMs with Dual Self-Rewards),通过自监督双奖励机制增强LMM的理解与生成能力。SUDER利用理解与生成任务间的内在对偶性,为彼此提供自监督优化信号。具体地,针对某一任务域的输入采样多个输出,再反转输入-输出对,在模型内计算双似然作为自奖励进行优化。大量实验结果表明,该方法可在无外部监督下有效提升模型性能,尤其在文本到图像任务中表现突出。
原文摘要 · Abstract (English)
Building upon large language models (LLMs), recent large multimodal models (LMMs) unify cross-model understanding and generation into a single framework. However, LMMs still struggle to achieve accurate vision-language alignment, prone to generating text responses contradicting the visual input or failing to follow the text-to-image prompts. Current solutions require external supervision (e.g., human feedback or reward models) and only address unidirectional tasks-either understanding or generation. In this work, based on the observation that understanding and generation are naturally inverse dual tasks, we propose \textbf{SUDER} (\textbf{S}elf-improving \textbf{U}nified LMMs with \textbf{D}ual s\textbf{E}lf-\textbf{R}ewards), a framework reinforcing the understanding and generation capabilities of LMMs with a self-supervised dual reward mechanism. SUDER leverages the inherent duality between understanding and generation tasks to provide self-supervised optimization signals for each other. Specifically, we sample multiple outputs for a given input in one task domain, then reverse the input-output pairs to compute the dual likelihood within the model as self-rewards for optimization. Extensive experimental results on visual understanding and generation benchmarks demonstrate that our method can effectively enhance the performance of the model without any external supervision, especially achieving remarkable improvements in text-to-image tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。