MAPO通过动态调整优势函数，解决推理模型训练中的优势反转问题。
MAPO: Mixed Advantage Policy Optimization
- 根据轨迹确定性动态重加权优势函数
- 在多个基准上超越现有最优方法，提升推理性能
- 适合需要稳定策略优化的大型模型训练场景
近期基于基础模型的强化学习进展，如分组相对策略优化（GRPO），显著提升了基础模型在推理任务上的表现。优势函数在GRPO中作为核心机制，用于排序轨迹重要性。然而，现有方法存在优势反转和优势镜像问题，阻碍了不同查询样本间的优势合理分配。本文提出一种简单但有效的GRPO策略——混合优势策略优化（MAPO）。我们发现轨迹出现具有不同确定性，并针对高确定性轨迹引入优势百分比偏差。进一步地，动态重加权不同轨迹确定性的样本的优势函数，从而自适应地配置优势函数以匹配样本特性。与相关最先进方法的对比及对不同优势变体的消融实验，验证了该方法的有效性。
原文摘要 · Abstract (English)
Recent advances in reinforcement learning for foundation models, such as Group Relative Policy Optimization (GRPO), have significantly improved the performance of foundation models on reasoning tasks. Notably, the advantage function serves as a central mechanism in GRPO for ranking the trajectory importance. However, existing explorations encounter both advantage reversion and advantage mirror problems, which hinder the reasonable advantage allocation across different query samples. In this work, we propose an easy but effective GRPO strategy, Mixed Advantage Policy Optimization (MAPO). We reveal that the trajectory appears with different certainty and propose the advantage percent deviation for samples with high-certainty trajectories. Furthermore, we dynamically reweight the advantage function for samples with varying trajectory certainty, thereby adaptively configuring the advantage function to account for sample-specific characteristics. Comparison with related state-of-the-art methods, along with ablation studies on different advantage variants, validates the effectiveness of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。