通过自适应难度调整，提升多模态模型强化学习的推理能力
DIVA-GRPO: Enhancing Multimodal Reasoning through Difficulty-Adaptive Variant Advantage
- 根据问题难度动态调整变体样本选择，优化奖励分布
- 在6个基准上实现更高推理准确率与训练效率
- 适合研究多模态大模型强化学习与复杂推理的学者
基于组相对策略优化（GRPO）的强化学习已成为提升多模态大语言模型（MLLMs）推理能力的常用方法。尽管GRPO可在无评判器情况下实现长链推理，但在难题上常面临奖励稀疏、在过于简单或困难的问题上组间奖励一致性导致优势消失的问题。现有方法（如样本扩展、选择性使用、间接奖励设计）难以保持组内奖励分布的足够方差以生成清晰优化信号。为此，我们提出DIVA-GRPO，一种基于难度自适应的变体优势方法，从全局视角调节变体难度分布。该方法动态评估问题难度，采样适配难度的变体，并采用难度加权归一化缩放，在局部与全局组间计算优势，缓解奖励稀疏与优势消失问题，同时提升训练稳定性。在六个推理基准上的大量实验表明，DIVA-GRPO在训练效率和推理性能上均优于现有方法。
原文摘要 · Abstract (English)
Reinforcement learning (RL) with group relative policy optimization (GRPO) has become a widely adopted approach for enhancing the reasoning capabilities of multimodal large language models (MLLMs). While GRPO enables long-chain reasoning without a critic, it often suffers from sparse rewards on difficult problems and advantage vanishing when group-level rewards are too consistent for overly easy or hard problems. Existing solutions (sample expansion, selective utilization, and indirect reward design) often fail to maintain enough variance in within-group reward distributions to yield clear optimization signals. To address this, we propose DIVA-GRPO, a difficulty-adaptive variant advantage method that adjusts variant difficulty distributions from a global perspective. DIVA-GRPO dynamically assesses problem difficulty, samples variants with appropriate difficulty levels, and calculates advantages across local and global groups using difficulty-weighted and normalized scaling. This alleviates reward sparsity and advantage vanishing while improving training stability. Extensive experiments on six reasoning benchmarks demonstrate that DIVA-GRPO outperforms existing approaches in training efficiency and reasoning performance. Code: https://github.com/Siaaaaaa1/DIVA-GRPO
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。