通过梯度对齐筛选高效推理数据，显著减少大模型强化学习所需训练样本。
LearnAlign: Data Selection for LLM Reinforcement Learning with Improved Gradient Alignment
- 基于成功率的可学性评估，智能挑选有潜力的训练数据。
- 在GSM8K上仅用少1000条数据，性能反超全量数据（77.5% vs 77.0%）。
- 适合追求高效微调的大模型研究者和资源受限场景使用者。
基于可验证奖励的强化学习（RLVR）已成为提升大语言模型推理能力的关键技术，但其数据效率低下仍是主要瓶颈。为此，我们提出一种基于梯度对齐的新方法LearnAlign，用于在RLVR后训练阶段智能筛选可学习且具代表性的推理数据。为克服梯度范数中的响应长度偏差问题，我们引入基于成功率的数据可学性指标，反映每个数据点的学习潜力。在五个推理基准上的实验表明，该方法显著降低训练数据需求，同时实现性能不降甚至超越全数据训练。具体而言，在GSM8K基准上，仅需减少1000个数据点即可获得更高性能（77.5%），优于全数据集表现（77.0%）。此外，在数学与代码任务中，使用更少的DAPO-MATH-17K数据也展现出高效性。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) has become a key technique for enhancing LLMs' reasoning abilities, yet its data inefficiency remains a major bottleneck. To address this critical yet challenging issue, we present a novel gradient-alignment-based method, named LearnAlign, which intelligently selects the learnable and representative training reasoning data for RLVR post-training. To overcome the well-known response-length bias in gradient norms, we introduce the data learnability based on the success rate, which indicates the learning potential of each data point. Experiments across five reasoning benchmarks show that our method significantly reduces training data requirements while achieving minor performance degradation or even improving performance compared to full-data training. Specifically, it reduces data requirements by up to 1,000 data points with better performance (77.5%) than that on the full dataset on the GSM8K benchmark (77.0%). Furthermore, its efficiency is demonstrated on both mathematical and code benchmarks by using much less data from the DAPO-MATH-17K dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。