用10万条短模型生成的思维链，让小模型也能长思考。
One Missing Piece for Open-Source Reasoning Models: A Dataset to Mitigate Cold-Starting Short CoT LLMs in RL
- 用现有短思维链模型构建10万条新思维链数据集
- 训练后模型推理能力接近原版大模型，强化强化学习效果2-3倍
- 适合想自研推理模型但缺乏大模型的开发者
随着R1公开发布，研究人员普遍通过直接蒸馏其长思维链（CoT）推理解答来训练新大型推理模型（LRM）。尽管已有研究证明可通过蒸馏复现LRM能力，但对已有模型（如R1）的持续依赖仍是该领域发展的关键瓶颈。为推动独立开发LRM的进程，本文探索构建不依赖推理时扩展的长CoT数据集的可能性。为此，我们提出Long CoT Collection数据集，包含10万条使用现有短CoT LLM生成的思维链注释。我们设计了一套流程，将o1模型的新推理策略引入短CoT LLM，使其能进行更长的思考，并实现对思维预算的可控管理，缓解过度思考问题。大量分析表明，该数据集质量与R1相当或略低。实验进一步验证，基于该数据集训练不仅能提升通用推理能力，还能显著增强强化学习表现：以该数据初始化的模型在RLVR中获得2-3倍的性能提升。
原文摘要 · Abstract (English)
With the release of R1, a publicly available large reasoning model (LRM), researchers commonly train new LRMs by training language models on R1's long chain-of-thought (CoT) inferences. While prior works show that LRMs' capabilities can be reproduced through direct distillation, the continued reliance on the existing models (e.g., R1) remains a critical limitation in advancing the field. As a first step toward independent LRM development, this paper explores the possibility of constructing a long CoT dataset with LLMs that are not trained for inference-time scaling. To this end, we present the Long CoT Collection, a dataset of 100K CoT rationales annotated using existing short CoT LLMs. We develop a pipeline that induces o1's novel reasoning strategies into short CoT LLMs, enabling them to think longer and introducing controllability over the thought budget to better manage the overthinking problem. Our extensive analyses validate that our dataset achieves quality comparable to--or slightly below--R1. Furthermore, our experiments demonstrate that training on our dataset not only strengthens general reasoning skills, but also provides a strong foundation for reinforcement learning--models initialized on our data achieve 2-3x larger gains with RLVR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。