用能量模型筛选强化学习训练数据,让模型专注攻克难题。
Learning-Zone Energy: Online Data Selection for Efficient RL Post-Training

- 基于三重信号构建在线数据筛选得分,聚焦模型真正需要学习的题目。
- 仅用40%数据量达到全量训练效果,训练量减少36%且跨域表现提升超45%。
- 适合追求高效微调与数学推理增强的LLM研究者和开发者。
强化学习后训练已成为激发大语言模型数学推理能力的主要范式,但现有方法如GRPO和DAPO在提示词间均匀分配采样与梯度预算,浪费算力于已掌握或远超当前能力的样本。为此,我们提出学习区能量(Learning-Zone Energy, LZE),一个理论严谨、完全在线的数据选择框架,将计算集中在模型的主动学习前沿。核心是定义一个闭式学习区能量评分,融合初始难度锚点、归一化结果不确定性项与通过率动量三个互补信号,形成单一标量,该标量可证明与组内相对策略梯度更新的期望幅度对齐。结合前向剪枝与回放机制,跳过已持续解决的提示词采样,定期检查遗忘现象,进一步降低实际耗时。在Qwen系列模型(1.5B-8B)上评估,覆盖GSM8K、MATH及DAPO-MATH数据集,每步仅保留40%训练数据即可匹配或超越全数据基线,尤其在AIME25上实现+45.9%、AMC23上+18.2%的显著跨域增益,同时估算训练浮点运算量减少36%。代码已开源:https://github.com/Stellaris167/LZE。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) post-training has emerged as the dominant paradigm for eliciting mathematical reasoning in Large Language Models (LLMs), yet prevailing techniques such as GRPO and DAPO distribute rollout and gradient budgets nearly uniformly across prompts, squandering compute on samples that are already mastered or remain far beyond the model's current capability. To address this fundamental inefficiency, we propose Learning-Zone Energy (LZE), a theoretically grounded, fully online data selection framework that concentrates computation on the model's active learning frontier. At its core, we define a closed-form Learning-Zone Energy Score that fuses three complementary signals, an initial-difficulty anchor, a normalized outcome-uncertainty term, and a pass-rate momentum, into a single scalar that is provably aligned with the expected magnitude of group-relative policy gradient updates. A forward pruner with replay further reduces wall-clock time cost by skipping rollout generation for persistently solved prompts while periodically checking for forgetting. Evaluated on Qwen-family models (1.5B-8B) across GSM8K, MATH and DAPO-MATH, our method retains only 40% of the training data per step yet matches or surpasses full-data baselines, with especially pronounced out-of-distribution gains on AIME25 (+45.9%) and AMC23 (+18.2%), alongside an estimated 36% reduction in training FLOPs. Our code is available at https://github.com/Stellaris167/LZE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。