用智能筛选方法从海量数据中挑出最有价值的推理题,大幅降低训练成本。
Select2Reason: Efficient Instruction-Tuning Data Selection for Long-CoT Reasoning
- 基于问题难度和推理路径长度,加权筛选高质量长链推理数据。
- 仅用10%精选数据,模型性能媲美甚至超过全量数据训练。
- 适合需要高效微调数学推理模型的研究者与工程师。
在预训练大语言模型中激活长链思维推理能力的一种实用方法是使用强推理模型(如DeepSeek-R1)生成的指令数据进行监督微调,这比强化学习更具成本效益。然而,包含超过10万样本的大规模指令集会带来显著训练开销,而针对长链思维指令的自动筛选策略仍不成熟。本文提出Select2Reason,一种面向长链思维推理的高效指令数据选择框架。从自修正、回溯等重思行为的涌现角度出发,研究影响长链思维指令质量的常见指标。Select2Reason利用量化器估计问题难度,并结合推理轨迹长度的启发式策略,通过加权排名优先选取高价值样本。在OpenR1-Math-220k上的实验证明,仅使用10%由Select2Reason选出的数据进行微调,模型在三个竞赛级和六个综合数学基准测试中的表现,可媲美或优于全量数据微调及开源基线OpenR1-Qwen-7B。进一步实验显示其在不同数据规模下的可扩展性、推理效率以及对其他指令池的低成本适应能力。
原文摘要 · Abstract (English)
A practical approach to activate long chain-of-thoughts reasoning ability in pre-trained large language models is to perform supervised fine-tuning on instruction datasets synthesized by strong Large Reasoning Models such as DeepSeek-R1, offering a cost-effective alternative to reinforcement learning. However, large-scale instruction sets with more than 100k samples incur significant training overhead, while effective strategies for automatic long-CoT instruction selection still remain unexplored. In this work, we propose Select2Reason, a novel and efficient instruction-tuning data selection framework for long-CoT reasoning. From the perspective of emergence of rethinking behaviors like self-correction and backtracking, we investigate common metrics that may determine the quality of long-CoT reasoning instructions. Select2Reason leverages a quantifier to estimate difficulty of question and jointly incorporates a reasoning trace length-based heuristic through a weighted scheme for ranking to prioritize high-utility examples. Empirical results on OpenR1-Math-220k demonstrate that fine-tuning LLM on only 10% of the data selected by Select2Reason achieves performance competitive with or superior to full-data tuning and open-source baseline OpenR1-Qwen-7B across three competition-level and six comprehensive mathematical benchmarks. Further experiments highlight the scalability in varying data size, efficiency during inference, and its adaptability to other instruction pools with minimal cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。