用进展增强的课程机制,让大模型多任务强化学习更高效。
PAC: Progress-Augmented Advantage Curriculum for Multi-Task Reinforcement Learning of LLMs

- 结合策略更新力度与实际奖励提升,动态分配训练资源。
- 在多层级和多领域推理任务中,用更少步数达到更好效果。
- 适合追求高效微调的大模型研究者与工程团队。
强化学习(RL)被用于提升大语言模型(LLMs)的推理能力,但训练数据涵盖多种异构任务。然而,多数RL后训练流程依赖固定或手动设计的任务混合方式,而任务的有效性会随训练进程变化。在线课程方法通常以更新幅度衡量可学习性,却忽略了更新是否带来实际奖励提升,可能导致采样预算被分配到无效任务上。我们提出PAC:一种面向多任务强化学习的进展增强优势课程方法,融合两种任务级信号:基于优势的可学习性(衡量任务能引发的策略更新幅度)与近期奖励增益(反映更新是否提升了任务性能)。采用贝叶斯汤普森采样控制器,在GRPO训练中动态分配滚动生成资源。我们在两个场景下评估:多层次推理与多领域推理。PAC显著提升样本效率与最终表现:以更少的滚动生成相当的验证得分,并在两个设置中均优于随机采样与基于优势的课程基线。结果表明,同时追踪优势信号与实际奖励增益,可构建有效的在线课程机制,适用于大模型后训练。
原文摘要 · Abstract (English)
Reinforcement learning (RL) is used to improve the reasoning abilities of LLMs, while training data span heterogeneous tasks. However, most RL post-training pipelines rely on fixed or manually designed task mixtures, even though task usefulness changes as training progresses. Online curriculum methods often define learnability by update magnitude, ignoring whether the update translates into reward gains, which can misallocate rollout budget toward tasks with large but ineffective updates. We propose PAC, a Progress-Augmented Advantage Curriculum for multi-task RL of LLMs that combines two task-level signals: advantage-derived learnability, which measures the magnitude of the policy update a task can induce, and recent reward gains, which show whether those updates have improved task performance. A Bayesian Thompson Sampling controller uses these signals to allocate rollouts across tasks during GRPO training. We evaluate PAC under two settings: a multi-level reasoning setting and a multi-domain reasoning setting. PAC improves sample efficiency and final performance: it reaches comparable validation scores with fewer rollout steps and achieves higher final averages than random sampling and advantage-based curriculum baselines in both settings. These results show that jointly tracking advantage signals and actual reward gains yields an effective online curriculum for LLM post-training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。