提出自适应策略优化方法，提升推理大模型训练稳定性和推理效率
Adaptive Group Policy Optimization: Towards Stable Training and Token-Efficient Reasoning
- 引入自适应损失函数缓解优势估计方差为零问题
- 训练更稳定，推理步骤用 token 数显著减少
- 适合追求高效推理与鲁棒训练的模型开发者
自DeepSeek-R1推出以来，组相对策略优化（GRPO）已成为训练推理类大模型的核心方法。然而我们发现其存在影响强化学习训练稳定性与推理效率的缺陷，例如优势估计方差为零。为此，我们提出自适应组策略优化（AGPO），采用一种简单但有效的方法——自适应损失函数，以缓解训练波动和推理令牌低效问题。实验表明，该方法在保持优异性能的同时，实现了更稳定的训练过程，并显著减少了推理步骤中的令牌消耗。
原文摘要 · Abstract (English)
Since DeepSeek-R1 popularized, Group Relative Policy Optimization (GRPO) has become the core part of training Reasoning LLMs. However, we find some deficiency that influences RL stability and inference efficiency, like zero-variance in advantage estimation. Thus, we propose Adaptive Group Policy Optimization (AGPO) which uses a simple but effective method, an adaptive loss function, to mitigate training fluctuation and token inefficiency. The experiments demonstrate our method achieves more stable training and superior performance with significantly fewer tokens in reasoning steps.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。