GAPO通过自适应选择信噪比最优区间，提升真实代码编辑场景下强化学习的鲁棒性。
GAPO: Robust Advantage Estimation for Real-World Code LLMs
- 基于信号噪声比自适应筛选每条指令的最优数据区间，用中位数替代均值作为优势估计基准
- 在9个3B-14B模型上实现最高5.30的域外准确率提升，且剪裁比例更低、显存效率更高
- 专为带噪声的真实代码编辑任务设计，适合需要高鲁棒性的代码生成与优化场景
强化学习广泛用于代码编辑类大语言模型的后训练，其中群体相对方法（如GRPO）因无需价值函数且能归一化优势估计而流行。但在真实代码编辑场景中，奖励分布常呈偏斜且含不可预测噪声，导致优势计算失真并引发大量回滚异常。为此，本文提出组自适应策略优化（GAPO），针对每条指令自适应选取信噪比最高的区间，以该区间的中位数作为自适应Q值，替代原群体均值进行优势计算，进一步抑制噪声。该方法具备良好鲁棒性，同时保持即插即用和高效特性。我们在涵盖10种编程语言的51,844个真实历史感知代码编辑任务上评估了九个指令微调的LLM（3B-14B）。GAPO相比GRPO及其变体DAPO，在域内（ID）最多提升4.35分，域外（OOD）最多提升5.30分，且剪裁比率更低、GPU吞吐更高。
原文摘要 · Abstract (English)
Reinforcement learning (RL) is widely used for post-training large language models (LLMs) in code editing, where group-relative methods, such as GRPO, are popular due to their critic-free and normalized advantage estimation. However, in real-world code-editing scenarios, reward distributions are often skewed with unpredictable noise, leading to distorted advantage computation and increased rollout outliers. To address this issue, we propose Group Adaptive Policy Optimization (GAPO), which adaptively finds an interval with the highest SNR (Signal to Noise Ratio) per prompt and uses the median of that interval as an adaptive Q to replace the group mean in advantage calculation to reduce noise further. This adaptive Q robustly handles rollout noise while remaining plug-and-play and efficient. We evaluate GAPO on nine instruction-tuned LLMs (3B-14B) using a collected large dataset of 51,844 real-world, history-aware code-editing tasks spanning 10 programming languages. GAPO yields up to 4.35 in-domain (ID) and 5.30 out-of-domain (OOD) exact-match improvements over GRPO and its variant DAPO, while achieving lower clipping ratios and higher GPU throughput. Code: https://github.com/TsingZ0/verl-GAPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。