提升多模态大模型推理能力,解决训练数据利用不足与文本依赖问题。
Boosting MLLM Reasoning with Text-Debiased Hint-GRPO
- 通过自适应提示增强难样本的数据利用效率
- 测试时校准文本预测逻辑,缓解模型对图像信息的依赖偏差
- 在11个数据集上显著超越现有方法,适合复杂推理任务
多模态大模型(MLLM)推理因其强大的问题求解能力受到广泛关注。当前方法分为两类:监督中间推理步骤的PRM和监督最终结果的ORM。近期DeepSeek-R1挑战了传统观点,证明基于ORM的GRPO方法具备优异泛化能力。然而,现有MLLM的GRPO算法在处理复杂多模态推理任务(如数学推理)时仍表现不佳。本文揭示两大瓶颈:低数据利用率(难以从难题中获得正向奖励)和文本偏见(训练后模型仅依赖文本生成)。为此,提出Hint-GRPO,通过自适应提供提示提升不同难度样本的数据利用;引入测试时图像条件校准机制,缓解文本偏见。在三个基础MLLM上跨十一项数据集的实验表明,所提方法大幅增强原模型推理能力,性能优于现有方法。代码已开源。
原文摘要 · Abstract (English)
MLLM reasoning has drawn widespread research for its excellent problem-solving capability. Current reasoning methods fall into two types: PRM, which supervises the intermediate reasoning steps, and ORM, which supervises the final results. Recently, DeepSeek-R1 has challenged the traditional view that PRM outperforms ORM, which demonstrates strong generalization performance using an ORM method (i.e., GRPO). However, current MLLM's GRPO algorithms still struggle to handle challenging and complex multimodal reasoning tasks (e.g., mathematical reasoning). In this work, we reveal two problems that impede the performance of GRPO on the MLLM: Low data utilization and Text-bias. Low data utilization refers to that GRPO cannot acquire positive rewards to update the MLLM on difficult samples, and text-bias is a phenomenon that the MLLM bypasses image condition and solely relies on text condition for generation after GRPO training. To tackle these problems, this work proposes Hint-GRPO that improves data utilization by adaptively providing hints for samples of varying difficulty, and text-bias calibration that mitigates text-bias by calibrating the token prediction logits with image condition in test-time. Experiment results on three base MLLMs across eleven datasets demonstrate that our proposed methods advance the reasoning capability of original MLLM by a large margin, exhibiting superior performance to existing MLLM reasoning methods. Our code is available at https://github.com/hqhQAQ/Hint-GRPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。