用奖励方差动态调节题目难度，让大模型循序渐进提升解题能力。
VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models
- 根据多轮推理奖励的方差判断题目难易，自动筛选适中难度样本
- 在五个数学基准上优于现有强化学习方法，提升模型推理准确率
- 适合需要分步训练的数学推理任务，尤其对初学者友好
基于策略的强化学习在提升大语言模型（LLM）数学推理能力方面发挥重要作用。然而，现有的基于回溯的强化学习方法（如GRPO、DAPO、GSPO等）未能显式考虑不同难度样本对LLM学习能力的影响，这与人类从简单到复杂的认知过程相悖。我们发现，回溯群体奖励的方差在一定程度上反映了当前样本对LLM的难度：过于简单或过于困难的样本方差较低，而中等难度样本方差较高。基于此，我们提出VCRL——一种基于方差的课程强化学习框架，通过奖励方差动态调控训练样本的难度。在五个数学基准和两种模型上的实验表明，VCRL显著优于当前主流的LLM强化学习基线方法。
原文摘要 · Abstract (English)
Policy-based reinforcement learning currently plays an important role in improving LLMs on mathematical reasoning tasks. However, existing rollout-based reinforcement learning methods (GRPO, DAPO, GSPO, etc.) fail to explicitly consider LLMs' learning ability for samples of different difficulty levels, which is contrary to the human cognitive process of mathematical reasoning tasks from easy to difficult. Intuitively, we find that the variance of the rollout group's reward in RLVR partly reflects the difficulty of the current sample for LLMs. Samples that are too easy or too difficult have a lower variance, while samples with moderate difficulty have a higher variance. Based on this, we propose VCRL, a curriculum reinforcement learning framework that dynamically controls the difficulty of training samples based on the variance of group rewards. Experiments on five mathematical benchmarks and two models reveal the advantages of VCRL over the current LLM RL baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。