arXiv:2510.01037cs.LGcs.AI2025-10被引 16

通过梯度分析优化提示选择与采样分配,提升大模型推理训练效率。

CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMs

  • 从强化学习梯度视角分析提示选择与采样分配对训练效率的影响。
  • 在8个数学推理基准上平均超越前人方法2.12点,7B模型提升4.82点。
  • 基于贝叶斯后验估计减少计算开销,适合高效训练推理类大模型。

课程学习在提升大语言模型(LLMs)在推理任务上的训练效率方面起着关键作用。然而,现有方法往往未能充分考虑提示难度的差异,或依赖简单的筛选机制在狭窄范围内选择提示数据集,导致显著的计算浪费。本文从强化学习梯度优化的角度出发,系统性地理论分析如何提升LLM的训练效率。我们识别出两个关键影响因素:训练提示的选择以及不同提示间采样数量的分配。理论分析表明,提示的采样分布决定梯度下降的收敛速度,而采样数量的分配影响整体梯度更新的一致性与稳定性。基于这些洞察,我们提出CurES,一种高效的训练方法,可加速收敛并利用贝叶斯后验估计最小化计算开销。实验表明,CurES在1.5B和7B模型上分别优于组相对策略优化(GRPO)+3.30点和+4.82点,在八个数学推理基准上平均超越最优先前样本高效方法+2.12点。此外,相比基线(包括GRPO),CurES表现出更快的收敛速度。

原文摘要 · Abstract (English)

Curriculum learning plays a crucial role in enhancing the training efficiency of large language models (LLMs) on reasoning tasks. However, existing methods often fail to adequately account for variations in prompt difficulty or rely on simplistic filtering mechanisms to select prompt datasets within a narrow criterion range, resulting in significant computational waste. In this work, we approach the problem from the perspective of reinforcement learning gradient optimization, offering a systematic and theoretical investigation into how to improve the training efficiency of LLMs. We identify two key factors influencing training efficiency: the selection of training prompts and the allocation of rollout quantities across different prompts. Our theoretical analysis reveals that the sampling distribution of prompts dictates the convergence rate of gradient descent, while the allocation of the rollout quantity influences the consistency and stability of overall gradient updates. Based on these insights, we propose CurES, an efficient training method that accelerates convergence and employs Bayesian posterior estimation to minimize computational overhead. Experiments demonstrate that our CurES outperforms Group Relative Policy Optimization (GRPO) by +3.30 points and +4.82 points with 1.5B and 7B models, respectively, and exceeds the best prior sample efficient methods by +2.12 points on average across eight math reasoning benchmarks. Additionally, CurES exhibits faster convergence compared to baselines, including GRPO.

课程学习大模型训练推理增强强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。