通过不对称优化提升多模态大模型的推理能力,解决过度思考和性能下降问题。
APO: Enhancing Reasoning Ability of MLLMs via Asymmetric Policy Optimization
- 采用分组策略,对正负样本分别动态调整难度和复杂度约束。
- 在多个推理基准上提升7%性能,超越7-11B级大模型。
- 保持通用任务表现,适合追求强推理与稳定性的研究者使用。
多模态大语言模型(MLLMs)虽能整合多种数据,但在复杂推理上仍表现不足。尽管强化学习(RL)可提升语言模型推理能力,但应用于MLLMs时面临性能下降和过度详细推理(过思)等问题。本文研究了KL惩罚与过思对RL训练的影响,提出不对称策略优化(APO),将采样响应分为正负样本。对正样本引入难度自适应发散塑造(DADS),动态调节KL权重以避免策略熵骤降,提升训练稳定性并保留原有知识。对负样本提出次优轨迹复杂度正则化(STCR),惩罚过长输出,缓解过思并促进简洁推理,同时保留探索能力。方法应用于Qwen2.5-VL-3B,构建View-R1-3B。该模型在多个推理基准上平均提升7%,优于7-11B级大型模型。更重要的是,其通用任务表现未退化,展现出更优泛化性。结果验证了DADS与STCR在提升复杂多模态推理中的有效性与普适性。代码将在https://github.com/Indolent-Kawhi/View-R1公开。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) are powerful at integrating diverse data, but they often struggle with complex reasoning. While Reinforcement learning (RL) can boost reasoning in LLMs, applying it to MLLMs is tricky. Common issues include a drop in performance on general tasks and the generation of overly detailed or "overthinking" reasoning. Our work investigates how the KL penalty and overthinking affect RL training in MLLMs. We propose Asymmetric Policy Optimization (APO) to address these issues, which divides the sampled responses into positive and negative groups. For positive samples, Difficulty-Adaptive Divergence Shaping (DADS) is introduced to dynamically adjust the KL divergence weight based on their difficulty. This method prevents policy entropy from dropping sharply, improves training stability, utilizes samples better, and preserves the model's existing knowledge. For negative samples, Suboptimal Trajectory Complexity Regularization (STCR) is proposed to penalize overly long responses. This helps mitigate overthinking and encourages more concise reasoning while preserving the model's explorative capacity. We apply our method to Qwen2.5-VL-3B, creating View-R1-3B. View-R1-3B significantly enhances reasoning capabilities, showing an average 7\% gain over the base model and outperforming larger MLLMs (7-11B) on various reasoning benchmarks. Importantly, unlike other reasoning-tuned MLLMs that often degrade on general tasks, View-R1-3B maintains consistent improvement, demonstrating superior generalization. These results highlight the effectiveness and broad applicability of our DADS and STCR techniques for advancing complex multimodal reasoning in MLLMs. The code will be made available at https://github.com/Indolent-Kawhi/View-R1.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。