AGPO动态调节大模型强化学习中的更新幅度和探索强度，提升推理性能。
AGPO: Adaptive Group Policy Optimization with Dual Statistical Feedback
- 基于群体统计信息自适应调整更新范围和解码温度
- 在9个数学/理工类任务上超越PPO/GRPO，GSM8K达67.3%
- 无需判别器，适用于多种大模型，代码开源
强化学习可提升大模型推理能力，但传统PPO/GRPO采用固定裁剪值和解码温度，导致训练脆弱且调参复杂。本文提出自适应组策略优化（AGPO），一种无需判别器的GRPO改进方法，利用群体级统计信息同时控制更新幅度与探索程度。AGPO通过共享探针推导的统计状态驱动两个控制器：(i) 自适应裁剪，根据奖励分布方差、偏度、探针投票熵、策略熵及逐步KL漂移设定信任区域大小；(ii) 双向自适应温度采样，基于相对运行基线的中心化不确定性，在基础温度上下加热或冷却解码。在九个中英文数学/理工类基准测试中，使用相同生成令牌预算的Qwen2.5-14B经AGPO训练后，在GSM8K上达到67.3%准确率，在MATH上达到40.5%。性能增益同样适用于Llama-3-8B与Gemma-2-9B，消融实验验证两模块互补性。实现代码已公开于https://github.com/wandugu/paper_agpo。
原文摘要 · Abstract (English)
Reinforcement learning improves LLM reasoning, but PPO/GRPO typically use fixed clipping and decoding temperature, which makes training brittle and tuning-heavy. We propose Adaptive Group Policy Optimization (AGPO), a critic-free refinement of GRPO that uses group-level statistics to control both update magnitude and exploration. AGPO uses a shared probe-derived statistical state to drive two controllers: (i) adaptive clipping, which sets the trust-region size from reward dispersion and skewness, probe vote entropy, policy entropy, and step-wise KL drift; and (ii) bidirectional adaptive temperature sampling, which heats or cools decoding around a base temperature according to centered uncertainty relative to a running baseline. On nine English and Chinese math/STEM benchmarks, Qwen2.5-14B trained with AGPO outperforms PPO/GRPO under the same generated-token budget, reaching 67.3% on GSM8K and 40.5% on MATH. Gains transfer to Llama-3-8B and Gemma-2-9B, and ablations confirm both modules are complementary. Our implementation is publicly available at https://github.com/wandugu/paper_agpo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。