模型自动生成图像并反哺训练,实现图文能力协同提升。
UniRL: Self-Improving Unified Multimodal Models via Supervised and Reinforcement Learning
- 用自身生成的图像作为训练数据,无需外部图像
- 图文任务性能同步提升,生成与理解更均衡
- 仅需少量额外训练步骤,适合快速迭代优化
统一多模态大模型如Show-o和Janus在生成与理解任务中表现优异,但通常依赖大规模数据集和高计算开销。现有后训练方法常依赖外部数据或仅限特定任务。本文提出UniRL,一种自改进的后训练方法:模型根据提示生成图像,并将其用于理解任务,同时利用理解结果监督生成过程,实现双向增强。采用监督微调(SFT)和组相对策略优化(GRPO)进行优化。UniRL三大优势:(1)无需外部图像数据,所有训练样本由模型自动生成;(2)同时提升生成与理解性能,缓解任务不平衡;(3)后训练阶段仅需数步额外训练。在Show-o和Janus上评估,分别获得0.77和0.65的GenEval得分。代码与模型将开源于https://github.com/showlab/UniRL。
原文摘要 · Abstract (English)
Unified multimodal large language models such as Show-o and Janus have achieved strong performance across both generation and understanding tasks. However, these models typically rely on large-scale datasets and require substantial computation during the pretraining stage. In addition, several post-training methods have been proposed, but they often depend on external data or are limited to task-specific customization. In this work, we introduce UniRL, a self-improving post-training approach. Our approach enables the model to generate images from prompts and use them as training data in each iteration, without relying on any external image data. Moreover, it enables the two tasks to enhance each other: the generated images are used for understanding, and the understanding results are used to supervise generation. We explore supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO) to optimize the models. UniRL offers three key advantages: (1) it requires no external image data, as all training samples are generated by the model itself during training; (2) it not only improves individual task performance, but also reduces the imbalance between generation and understanding; and (3) it requires only several additional training steps during the post-training stage. We evaluate UniRL on top of Show-o and Janus, achieving a GenEval score of 0.77 for Show-o and 0.65 for Janus. Code and models will be released in https://github.com/showlab/UniRL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。