让大模型训练更高效：挑出难易不定的题优先练。
Learning to Reason at the Frontier of Learnability
- 选成功概率波动大的题目优先训练，提升学习信号质量。
- 在多个算法和数据集上，性能显著优于传统方法。
- 适合做数学推理等需要强化学习的LLM训练场景。
强化学习现被广泛用于大语言模型训练的最后阶段，尤其针对数学等推理类任务。通常模型在每个训练步骤中反复尝试同一问题，从成功与失败中学习。然而我们发现，在PPO和VinePPO两种主流算法及两个常用数据集上，大量题目要么全被解决（已掌握），要么全未解决（无学习信号）。为解决此问题，我们引入强化学习文献中的「可学习性采样」方法，应用于大模型强化学习阶段。该课程设计优先选择成功率波动高的题目，即模型时而成功、时而失败的问题。实验表明，该策略在多种算法与数据集上均能持续提升训练表现，为大模型强化学习的效率与效果提供了新路径。
原文摘要 · Abstract (English)
Reinforcement learning is now widely adopted as the final stage of large language model training, especially for reasoning-style tasks such as maths problems. Typically, models attempt each question many times during a single training step and attempt to learn from their successes and failures. However, we demonstrate that throughout training with two popular algorithms (PPO and VinePPO) on two widely used datasets, many questions are either solved by all attempts - meaning they are already learned - or by none - providing no meaningful training signal. To address this, we adapt a method from the reinforcement learning literature - sampling for learnability - and apply it to the reinforcement learning stage of LLM training. Our curriculum prioritises questions with high variance of success, i.e. those where the agent sometimes succeeds, but not always. Our findings demonstrate that this curriculum consistently boosts training performance across multiple algorithms and datasets, paving the way for more efficient and effective reinforcement learning with LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。