提出AAPO算法，提升大模型推理能力。
AAPO: Enhancing the Reasoning Capabilities of LLMs with Advantage Margin
- 用基于优势差距的估计增强交叉熵损失优化
- 在多个数学推理基准上表现优于现有方法
- 适合需要高效强化学习训练的模型改进场景
强化学习（RL）已成为提升大语言模型（LLMs）推理能力的有效方法，尤其在链式思维（CoT）数据有限时，监督微调（SFT）效果不佳。基于群体相对优势估计的方法（如GRPO）因无需价值模型而简化训练，但现有方法在优势估计趋近零时仍存在训练效率问题。为此，本文提出优势增强策略优化（AAPO），通过基于差距的估计增强优势，优化交叉熵损失，有效缓解了该问题。在多个数学推理基准上的实验表明，AAPO性能显著优于现有方法。代码已开源：https://github.com/JianxXiong/AAPO。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has emerged as an effective approach for enhancing the reasoning capabilities of large language models (LLMs), especially in scenarios where supervised fine-tuning (SFT) falls short due to limited chain-of-thought (CoT) data. Among RL-based post-training methods, group relative advantage estimation, as exemplified by Group Relative Policy Optimization (GRPO), has attracted considerable attention for eliminating the dependency on the value model, thereby simplifying training compared to traditional approaches like Proximal Policy Optimization (PPO). However, we observe that exsiting group relative advantage estimation method still suffers from training inefficiencies, particularly when the estimated advantage approaches zero. To address this limitation, we propose Advantage-Augmented Policy Optimization (AAPO), a novel RL algorithm that optimizes the cross-entropy (CE) loss using advantages enhanced through a margin-based estimation scheme. This approach effectively mitigates the inefficiencies associated with group relative advantage estimation. Experimental results on multiple mathematical reasoning benchmarks demonstrate the superior performance of AAPO. Code is available at https://github.com/JianxXiong/AAPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。