解决多模态大模型推理中的奖励不稳定问题，提升准确率。
Enhancing Multi-Modal LLMs Reasoning via Difficulty-Aware Group Normalization
- 按样本难易度重新分组，组内统一标准化
- 在多个多模态推理基准上显著提升性能
- 适合需要稳定训练的多模态大模型研究者
基于可验证奖励的强化学习（RLVR）和组相对策略优化（GRPO）显著提升了大语言模型的推理能力。然而，将这些方法扩展到多模态场景时面临关键挑战：基于标准差的归一化容易被极端奖励样本（几乎正或负）扭曲。与纯文本大模型不同，多模态模型对这类干扰尤为敏感，因为感知误差和推理不确定性都会影响输出。为此，我们通过视觉熵衡量感知复杂度、用模型置信度捕捉推理不确定性，定义样本难易度。基于此，提出难度感知组归一化（Durian），按难易度重新分组样本，并在每组内共享标准差。该方法在保留组内差异的同时，消除对极端样本的敏感性，在多个多模态推理基准上取得显著提升。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) and Group Relative Policy Optimization (GRPO) have significantly advanced the reasoning capabilities of large language models. Extending these methods to multimodal settings, however, faces a critical challenge: the instability of std-based normalization, which is easily distorted by extreme samples with nearly positive or negative rewards. Unlike pure-text LLMs, multimodal models are particularly sensitive to such distortions, as both perceptual and reasoning errors influence their responses. To address this, we characterize each sample by its difficulty, defined through perceptual complexity (measured via visual entropy) and reasoning uncertainty (captured by model confidence). Building on this characterization, we propose difficulty-aware group normalization (Durian), which re-groups samples by difficulty levels and shares the std within each group. Our approach preserves GRPO's intra-group distinctions while eliminating sensitivity to extreme cases, yielding significant performance gains across multiple multimodal reasoning benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。