arXiv:2507.09850cs.AI2025-07中稿 · the Second AI for …被引 1

仅用20个高质量思维链样本,就能让基础模型获得强推理能力。

The Challenge of Teaching Reasoning to LLMs Without RL or Distillation

  • 用20个思维链示例轻量微调,激活基础模型推理能力。
  • 效果超越更大规模的专用数学模型,证明小样本高效。
  • 适合想低成本提升模型推理能力的研究者与开发者。

具备推理能力的语言模型通过生成长而明确的思维链(CoT)在复杂任务中表现卓越。尽管近期研究显示可通过强化学习或从更强模型(如DeepSeek-R1)蒸馏获得此类思维链,但已有工作表明,仅使用短思维链提示也能提升推理性能。本文探讨是否仅通过提示或极简微调即可在基础模型中诱导出长思维链。利用来自推理模型QwQ-32B-Preview的20个长思维链示例,对基础模型Qwen2.5-32B进行轻量微调,结果其性能超过更大的Qwen2.5-Math-72B-Instruct模型,证明少量高质量示例足以激发强大推理能力。进一步尝试使用非推理模型和人工标注的思维链数据,结合提示工程、多轮编辑与结构引导,但均未达到专家思维链的表现,说明专家思维链中存在难以复制的潜在特性。分析了问题难度、多样性与答案长度等关键因素对推理蒸馏的影响。尽管挑战仍在,我们仍认为精心设计的人工撰写思维链,即使数量极少,也能激活基础模型的推理行为。本文发布多阶段精炼的人类撰写的思维链数据集,邀请进一步探究小规模推理监督为何如此有效。

原文摘要 · Abstract (English)

Reasoning-capable language models achieve state-of-the-art performance in diverse complex tasks by generating long, explicit Chain-of-Thought (CoT) traces. While recent works show that base models can acquire such reasoning traces via reinforcement learning or distillation from stronger models like DeepSeek-R1, previous works demonstrate that even short CoT prompting without fine-tuning is able to improve reasoning. We ask whether long CoT can be induced in a base model using only prompting or minimal tuning. Using just 20 long CoT examples from the reasoning model \texttt{QwQ-32B-Preview}, we lightly fine-tune the base model \texttt{Qwen2.5-32B}. The resulting model outperforms the much larger \texttt{Qwen2.5-Math-72B-Instruct}, showing that a handful of high-quality examples can unlock strong reasoning capabilities. We further explore using CoT data from non-reasoning models and human annotators, enhanced with prompt engineering, multi-pass editing, and structural guidance. However, neither matches the performance of reasoning model traces, suggesting that certain latent qualities of expert CoT are difficult to replicate. We analyze key properties of reasoning data, such as problem difficulty, diversity, and answer length, that influence reasoning distillation. While challenges remain, we are optimistic that carefully curated human-written CoT, even in small quantities, can activate reasoning behaviors in base models. We release our human-authored dataset across refinement stages and invite further investigation into what makes small-scale reasoning supervision so effective.

推理生成小样本学习思维链模型微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。