arXiv:2606.26797cs.LG2026-06中稿 · ICML

用初始推理片段就能筛选高质量思维数据,效率更高。

Reasoning Quality Emerges Early: Data Curation for Reasoning Models

论文配图:Reasoning Quality Emerges Early: Data Curation for Reasoning Models
图 1 · 摘自论文原文
  • 仅凭前100个推理词的损失值判断题目难易
  • 在医疗和数学推理任务上提升1.7%准确率
  • 适合想高效训练推理模型的研究者

对少量高质量长推理链进行监督微调,是激发大语言模型强推理能力的有效方法。然而,现有数据筛选方法依赖强推理模型评估多样性和难度,成本高且质量不理想。本文发现,仅通过预训练模型在随机扰动检查点上对前100个推理词的损失值,即可可靠识别复杂问题。进一步证明,前1000个推理词在少数扰动检查点上的损失模式相似,可推断出相近梯度。我们在Qwen2.5-7B和Llama3.1-8B模型上对M23K医学推理与OpenThoughts-Math数据集进行实验,结果表明该方法相比基线最高提升1.7%,同时节省91%的令牌消耗。

原文摘要 · Abstract (English)

Supervised fine-tuning (SFT) on a small, high-quality set of long reasoning traces is an effective approach for eliciting strong reasoning capabilities in Large Language Models (LLMs). However, existing methods for curating high-quality SFT data rely heavily on strong reasoning models to filter examples based on diversity and difficulty, making the curation process costly while often yielding suboptimal data quality. In this work, we show that diverse and challenging reasoning examples can be identified using only the initial reasoning tokens. Specifically, we demonstrate that difficult problems can be reliably detected based on the loss of the first 100 reasoning tokens evaluated at a randomly perturbed checkpoint of the pretrained model. We further show that examples exhibiting similar loss patterns over their first 1k reasoning tokens across a small number of perturbed checkpoints extrapolating along the fine-tuning trajectory provably induce similar gradients. We validate our approach through extensive experiments on fine-tuning Qwen2.5-7B and Llama3.1-8B models on the M23K medical reasoning and OpenThoughts-Math datasets. Our method outperforms existing baselines by up to 1.7% while being 91% more token efficient.

推理模型数据筛选高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。