通过能力-难度对齐提升大模型推理强化学习采样效率
Rethinking the Sampling Criteria in Reinforcement Learning for LLM Reasoning: A Competence-Difficulty Alignment Perspective
- 基于历史表现差异聚合估计题目难度，稳定可靠
- 用固定点机制动态选择与模型能力匹配的题目
- 在多个数学基准上准确率最优，速度比现有方法快2.33倍
强化学习在提升大语言模型推理能力方面具有潜力，但其在推演阶段样本效率低。现有方法尝试通过按题目难度调度来提高效率，但难以准确估计难度，且忽视模型能力与题目难度的对齐，导致效果不佳。为此，本文提出能力-难度对齐采样（CDAS），通过聚合题目历史表现差异实现难度的精确稳定估计，并利用固定点系统量化模型能力，自适应选择与当前能力匹配的题目。在多个高难度数学基准上的实验表明，CDAS在准确率和效率上均显著优于基线方法，平均准确率最高，且相比动态采样（DAPO中较优策略）提速2.33倍。
原文摘要 · Abstract (English)
Reinforcement learning exhibits potential in enhancing the reasoning abilities of large language models, yet it is hard to scale for the low sample efficiency during the rollout phase. Existing methods attempt to improve efficiency by scheduling problems based on problem difficulties. However, these approaches suffer from unstable and biased estimations of problem difficulty and fail to capture the alignment between model competence and problem difficulty in RL training, leading to suboptimal results. To tackle these limitations, this paper introduces $\textbf{C}$ompetence-$\textbf{D}$ifficulty $\textbf{A}$lignment $\textbf{S}$ampling ($\textbf{CDAS}$), which enables accurate and stable estimation of problem difficulties by aggregating historical performance discrepancies of problems. Then the model competence is quantified to adaptively select problems whose difficulty is in alignment with the model's current competence using a fixed-point system. Experimental results across a range of challenging mathematical benchmarks show that CDAS achieves great improvements in both accuracy and efficiency. CDAS attains the highest average accuracy against baselines and exhibits significant speed advantages compared to Dynamic Sampling, a competitive strategy in DAPO, which is 2.33 times slower than CDAS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。