arXiv:2502.11886cs.LGcs.AI2025-02被引 139

只用1389个精选样本,就能让大模型推理能力超越8523个样本的训练集。

LIMR: Less is More for RL Scaling

论文配图:LIMR: Less is More for RL Scaling
图 1 · 摘自论文原文
  • 基于学习轨迹评估,自动筛选高价值训练样本。
  • 仅1389样本即达8523样本效果,AIME24准确率提升16.7%。
  • 适合追求高效推理增强的模型优化者使用。

本文探讨强化学习训练数据对大语言模型推理能力提升的真实影响。尽管o1、Deepseek R1和Kimi1.5等成果展示出强化学习潜力,但其训练数据需求不透明,阻碍了系统性进展。我们从基础模型出发,挑战‘数据量越大越好’的假设,发现仅用1,389个精心挑选的样本即可超越完整的8,523样本数据集。为此提出学习影响度量(LIM),通过评估样本与模型学习轨迹的契合度,实现高效样本筛选。实验表明,该方法在仅使用1,389样本时,性能与全量数据相当甚至更优。尤其在7B规模模型上,相较现有数据高效方法(如LIMO和s1),LIMR在AIME24上实现16.7%更高的准确率,在MATH500上分别领先13.0%和22.2%。研究揭示:精准样本选择比单纯扩增数据量更能激发模型推理潜能。为推动可复现研究,项目已开源,包含LIM实现、训练代码、评估脚本、精选数据集及训练模型,详见https://github.com/GAIR-NLP/LIMR。

原文摘要 · Abstract (English)

In this paper, we ask: what truly determines the effectiveness of RL training data for enhancing language models' reasoning capabilities? While recent advances like o1, Deepseek R1, and Kimi1.5 demonstrate RL's potential, the lack of transparency about training data requirements has hindered systematic progress. Starting directly from base models without distillation, we challenge the assumption that scaling up RL training data inherently improves performance. we demonstrate that a strategically selected subset of just 1,389 samples can outperform the full 8,523-sample dataset. We introduce Learning Impact Measurement (LIM), an automated method to evaluate and prioritize training samples based on their alignment with model learning trajectories, enabling efficient resource utilization and scalable implementation. Our method achieves comparable or even superior performance using only 1,389 samples versus the full 8,523 samples dataset. Notably, while recent data-efficient approaches (e.g., LIMO and s1) show promise with 32B-scale models, we find it significantly underperforms at 7B-scale through supervised fine-tuning (SFT). In contrast, our RL-based LIMR achieves 16.7% higher accuracy on AIME24 and outperforms LIMO and s1 by 13.0% and 22.2% on MATH500. These results fundamentally reshape our understanding of RL scaling in LLMs, demonstrating that precise sample selection, rather than data scale, may be the key to unlocking enhanced reasoning capabilities. For reproducible research and future innovation, we are open-sourcing LIMR, including implementation of LIM, training and evaluation code, curated datasets, and trained models at https://github.com/GAIR-NLP/LIMR.

强化学习数据效率推理增强样本筛选

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。