用智能筛选器动态选题,让大模型仅靠一次尝试就能更好解数学题。
Selector-Guided Autonomous Curriculum for One-Shot Reinforcement Learning from Verifiable Rewards

- 设计可学习的筛选模型,综合成功率、奖励方差、输出分歧和难度四维特征选题。
- 在Hendrycks MATH测试集上达到68.0%准确率,优于现有最优方法的64.0%。
- 适合资源受限场景下提升大模型数学推理能力,尤其适用于少样本强化学习。
近期,基于可验证奖励的强化学习(RLVR)已成为提升大语言模型数学推理能力的有效方法,仅需单个示例即可。当前最先进的1次性RLVR模型依赖历史奖励方差等启发式规则选择样本,但本文发现该指标对迁移价值的衡量存在根本偏差。为此,提出选择器引导的自主课程(SGAC)方法,构建一个可学习的选择器模型,其输入为多维特征空间:成功概率、奖励方差、输出分歧(熵)与语义难度。在候选问题池的实证评估中,发现输出分歧比奖励方差更能有效预测后续迭代中的推理提升。基于此,开发了自动课程算法,从大规模候选池中动态提取问题,按学习选择器排序,并执行1次性GRPO微爆发训练。实验使用Hendrycks MATH基准,以Qwen2.5-Math-1.5B为基线,本框架在留出数据集上获得68.0%准确率,优于现有最佳模型的64.0%,也高于Wang等人提出的1次性RLVR检查点(66.0%)。结果表明,在严重数据受限条件下,基于熵的智能数据筛选能带来显著推理性能提升。
原文摘要 · Abstract (English)
Recently, Reinforcement Learning from Verifiable Rewards (RLVR) has been established as a highly effective technique for augmenting the math reasoning skills of Large Language Models (LLMs) based on a single instance. Current state-of-the-art 1-shot RLVR models adopt heuristics for selecting instances, mostly based on historical variance in rewards, which we find to be inherently misleading as a measure of transferability value. In this paper, we propose a Selector-Guided Autonomous Curriculum (SGAC) approach, which employs a learnable selector model on a multi-dimensional feature space consisting of success probability, reward variance, output disagreement (entropy), and semantic difficulty level, instead of the static reward variance heuristic. In our empirical evaluation on pools of candidate problems, we observed that output disagreement, rather than reward variance, is the strongest predictor of reasoning gains in subsequent iterations. Leveraging this finding, we develop an autonomous curriculum algorithm for dynamically siphoning candidate problems from a large pool, ranking them by the learned selector, and running micro-bursts of 1-shot GRPO. Our framework is evaluated using the Hendrycks MATH benchmark, with the Qwen2.5-Math-1.5B model serving as the baseline. Our framework obtains an accuracy of 68.0\% on the hold-out dataset, which is better than the accuracy obtained from the state-of-the-art model, 64.0\%, as well as the 1-shot RLVR checkpoint proposed by Wang et al., which achieved an accuracy of 66.0\%. The results confirm that entropy-based intelligent data curation leads to strict reasoning improvement over static training methods, particularly in severely limited data conditions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。