arXiv:2605.11403cs.LGcs.AI2026-05

通过动态调整探索强度和题目难度分布,提升大模型数学推理的训练效率。

fg-expo: Frontier-guided exploration-prioritized policy optimization via adaptive kl and gaussian curriculum

  • 根据模型准确率动态调节KL惩罚,允许在困难时自由探索
  • 用高斯分布聚焦中等难度题目,获取更有效的梯度信号
  • 在多个基准上显著提升推理能力,尤其改善多步推理性能

强化学习结合可验证奖励(RLVR)已成为大语言模型数学推理的标准范式,其中组相对策略优化(GRPO)是主流算法。我们发现GRPO存在两个被忽视的低效问题:固定KL系数在模型需要大幅偏离参考策略时过度限制探索;均匀采样题目忽略了中等难度问题能提供最丰富的梯度信息。为此提出FG-ExPO(前沿引导的探索优先策略优化),集成两个轻量级组件。准确率条件化KL缩放(AKL)通过批次平均准确率的平滑非线性函数动态调整KL惩罚强度,模型表现差时放松约束,表现良好时加强控制。高斯课程采样(GCS)以约0.5的中等准确率为均值,按高斯分布分配题目采样权重,聚焦训练于模型的学习前沿。在DeepSeek-R1-Distill-Qwen-1.5B和Qwen3-8B-Base上对六个主流数学推理基准进行评估。结果表明,FG-ExPO持续优于原始GRPO,AIME 2025 pass@32指标绝对提升13.34,从63.33%增至76.67%,8B模型平均pass@32提升2.66。相比pass@1,pass@32提升更大,验证了FG-ExPO在固定推理预算下有效拓展了模型的探索空间。

原文摘要 · Abstract (English)

Reinforcement Learning with Verifiable Rewards (RLVR) has become the standard paradigm for LLM mathematical reasoning, with Group Relative Policy Optimization (GRPO) serving as the dominant algorithm. We identify two overlooked inefficiencies inherent in GRPO. First, a fixed KL coefficient overly restricts policy exploration at moments when the model needs to diverge significantly from the reference policy. Second, uniform question sampling overlooks that moderately difficult problems produce the most informative gradient signals. We propose FG-ExPO, short for Frontier-Guided Exploration-Prioritized Policy Optimization, which integrates two lightweight components. Accuracy-Conditioned KL Scaling (AKL) adjusts the KL penalty strength through a smooth nonlinear function of batch average accuracy, loosening the constraint when the model performs poorly and strengthening it when the model achieves satisfactory results. Gaussian Curriculum Sampling (GCS) assigns sampling weights to questions following a Gaussian distribution centered at a moderate accuracy level around 0.5, focusing model training on its learning frontier. We conduct evaluations on DeepSeek-R1-Distill-Qwen-1.5B and Qwen3-8B-Base across six mainstream mathematical reasoning benchmarks. Experimental results demonstrate that FG-ExPO consistently outperforms vanilla GRPO. It delivers an absolute improvement of 13.34 on the AIME 2025 pass@32 metric, rising from 63.33 percent to 76.67 percent, and obtains an average pass@32 gain of 2.66 on the 8B model. The substantially larger performance gains observed on pass@32 compared to pass@1 verify that FG-ExPO enlarges the model's effective exploration space under a fixed inference budget.

强化学习数学推理策略优化大模型训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。