动态调整提示长度,让大模型在最优难度区间高效学习推理。
Staying in the Sweet Spot: Responsive Reasoning Evolution via Capability-Adaptive Hint Scaffolding
- 根据模型能力实时调整提示长度,保持训练难度在高效区
- 在6个数学推理基准上比GRPO高11.8分,比SFT高10.5分
- 适合需要高效提升推理能力的LLM训练场景
基于可验证奖励的强化学习(RLVR)在提升大语言模型(LLM)推理能力方面取得显著进展。然而,现有RLVR方法常因训练数据难度与模型能力不匹配而导致探索效率低下:问题过难时模型无法发现有效推理路径,问题过易时则难以习得新能力。本文通过量化损失下降速度与回滚准确率的关系,正式分析了问题难度的影响。基于此,提出SEELE——一种监督辅助的RLVR框架,通过动态调整问题难度以维持在高效区域。SEELE为每个训练样本添加一个提示(完整解法的一部分),并针对每个问题自适应调整提示长度以达到最优难度。为确定最佳提示长度,它采用多轮回滚采样策略,每轮利用项目反应理论模型拟合前几轮收集的准确率-提示长度对,预测下一轮所需提示长度。这种实例级、实时的难度调节使问题难度与模型能力同步演化,显著提升探索效率。实验表明,SEELE在六个数学推理基准上的平均表现优于GRPO 11.8分、优于SFT 10.5分,且超越此前最优的监督辅助方法3.6分。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) has achieved remarkable success in enhancing the reasoning capabilities of large language models (LLMs). However, existing RLVR methods often suffer from exploration inefficiency due to mismatches between the training data's difficulty and the model's capability. LLMs fail to discover viable reasoning paths when problems are overly difficult, while learning little new capability when problems are too simple. In this work, we formalize the impact of problem difficulty by quantifying the relationship between loss descent speed and rollout accuracy. Building on this analysis, we propose SEELE, a novel supervision-aided RLVR framework that dynamically adjusts problem difficulty to stay within the high-efficiency region. SEELE augments each training sample by appending a hint (part of a full solution) after the original problem. Unlike previous hint-based approaches, SEELE deliberately and adaptively adjusts the hint length for each problem to achieve an optimal difficulty. To determine the optimal hint length, SEELE employs a multi-round rollout sampling strategy. In each round, it fits an item response theory model to the accuracy-hint pairs collected in preceding rounds to predict the required hint length for the next round. This instance-level, real-time difficulty adjustment aligns problem difficulty with the evolving model capability, thereby improving exploration efficiency. Experimental results show that SEELE outperforms Group Relative Policy Optimization (GRPO) and Supervised Fine-tuning (SFT) by +11.8 and +10.5 points, respectively, and surpasses the best previous supervision-aided approach by +3.6 points on average across six math reasoning benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。