统一多模态大模型推理能力训练,提升复杂任务表现
Mixed-R1: Unified Reward Perspective For Reasoning Capability in Multimodal Large Language Models
- 设计混合奖励函数与45K高质量数据集,覆盖多种多模态任务
- 在多个模型上实现推理能力显著提升,支持选择题、图表分析等任务
- 适合研究多模态推理、强化学习训练的学者和开发者
近期大语言模型通过强化学习展现出推理能力。尽管已有工作利用组相对策略优化(GRPO)对多模态大模型(MLLMs)进行后训练,但多数仅聚焦单一任务,如视觉定位、数学问题或图表分析。目前尚无方法能有效整合多源多模态任务实现稳定强化学习。本文提出Mixed-R1,一个统一且简洁的框架,包含混合奖励函数设计(Mixed-Reward)与混合后训练数据集(Mixed-45K)。首先构建数据引擎,筛选高质量样本构建45K后训练数据集;其次设计多任务混合奖励机制,包含四类:匹配奖励(用于二分类或多选题)、图表奖励(针对图表感知数据集)、IoU奖励(用于视觉定位任务)及开放文本奖励(针对长文本生成任务,如图像描述)。为处理长文本内容,提出双向最大平均相似度(BMAS)奖励,通过生成结果与真实答案的分词器嵌入匹配计算。大量实验表明,该方法在Qwen2.5-VL和Intern-VL等多种规模模型上均有效提升推理能力。代码与模型已开源。
原文摘要 · Abstract (English)
Recent works on large language models (LLMs) have successfully demonstrated the emergence of reasoning capabilities via reinforcement learning (RL). Although recent efforts leverage group relative policy optimization (GRPO) for MLLMs post-training, they constantly explore one specific aspect, such as grounding tasks, math problems, or chart analysis. There are no works that can leverage multi-source MLLM tasks for stable reinforcement learning. In this work, we present a unified perspective to solve this problem. We present Mixed-R1, a unified yet straightforward framework that contains a mixed reward function design (Mixed-Reward) and a mixed post-training dataset (Mixed-45K). We first design a data engine to select high-quality examples to build the Mixed-45K post-training dataset. Then, we present a Mixed-Reward design, which contains various reward functions for various MLLM tasks. In particular, it has four different reward functions: matching reward for binary answer or multiple-choice problems, chart reward for chart-aware datasets, IoU reward for grounding problems, and open-ended reward for long-form text responses such as caption datasets. To handle the various long-form text content, we propose a new open-ended reward named Bidirectional Max-Average Similarity (BMAS) by leveraging tokenizer embedding matching between the generated response and the ground truth. Extensive experiments show the effectiveness of our proposed method on various MLLMs, including Qwen2.5-VL and Intern-VL on various sizes. Our dataset and model are available at https://github.com/xushilin1/mixed-r1.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。