用强化学习统一提升多模态模型的理解与生成能力
Co-Reinforcement Learning for Unified Multimodal Understanding and Generation
- 提出协同强化学习框架，共享策略优化实现理解与生成同步提升
- 在3个文生图数据集上生成性能平均提升7%，9个跨模态理解任务提升23%
- 适合研究多模态大模型联合优化的开发者和研究人员参考
本文首次探索通过群体相对策略优化的强化学习（RL）方法，实现统一多模态大语言模型（ULMs）的理解与生成能力协同增强。系统性初步实验揭示了ULMs在共享策略优化框架下实现双重能力协同演化的巨大潜力。基于此，我们提出CoRL框架，包含统一强化学习阶段以实现联合优化，以及针对特定任务的精细化强化学习阶段。由此构建的ULM-R1模型，在三个文本到图像生成数据集上平均提升7%，在九个跨模态理解基准上平均提升23%。结果验证了CoRL的有效性，凸显强化学习在促进多任务协同与优化方面的显著优势。代码已开源：https://github.com/mm-vl/ULM-R1。
原文摘要 · Abstract (English)
This paper presents a pioneering exploration of reinforcement learning (RL) via group relative policy optimization for unified multimodal large language models (ULMs), aimed at simultaneously reinforcing generation and understanding capabilities. Through systematic pilot studies, we uncover the significant potential of ULMs to enable the synergistic co-evolution of dual capabilities within a shared policy optimization framework. Building on this insight, we introduce CoRL, a co-reinforcement learning framework comprising a unified RL stage for joint optimization and a refined RL stage for task-specific enhancement. With the proposed CoRL, our resulting model, ULM-R1, achieves average improvements of 7% on three text-to-image generation datasets and 23% on nine multimodal understanding benchmarks. These results demonstrate the effectiveness of CoRL and highlight the substantial benefit of reinforcement learning in facilitating cross-task synergy and optimization for ULMs. Code is available at https://github.com/mm-vl/ULM-R1.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。