小模型用长思维链训练反而变差,研究发现这是错误累积导致的。
Through the Valley: Path to Effective Long CoT Training for Small Language Models
- 通过控制数据量实验,发现小模型在长思维链上训练易出错并性能下降。
- 仅8000条长思维链数据就让部分模型性能下降75%,22万条也难恢复。
- 适合做小模型推理优化的研究者,尤其关注微调与强化学习的实践者。
长链式思维(Long CoT)监督已成为提升语言模型推理能力的常见策略。尽管对大模型有效,我们发现小语言模型(SLMs;≤3B参数)在有限长CoT数据上训练时会出现显著性能退化,称为“长思维链退化”(Long CoT Degradation)。在Qwen2.5、LLaMA3和Gemma3系列模型上广泛实验表明,该现象普遍存在。某些情况下,仅使用8000条长CoT样本训练后,模型性能相比微调前最高下降75%。更令人意外的是,部分极小模型即使使用22万条长CoT数据也无法恢复或超越原有性能。分析表明,该现象源于错误累积:虽然长响应提升多步推理能力,但也放大了错误传播风险。此外,我们观察到长CoT退化可能对下游强化学习产生负面影响,但可通过充分规模的监督微调(SFT)缓解。研究挑战了长CoT对小模型普遍有益的假设,为构建高效小规模推理模型提供实用指导。
原文摘要 · Abstract (English)
Long chain-of-thought (CoT) supervision has become a common strategy to enhance reasoning in language models. While effective for large models, we identify a phenomenon we call Long CoT Degradation, in which small language models (SLMs; <=3B parameters) trained on limited long CoT data experience significant performance deterioration. Through extensive experiments on the Qwen2.5, LLaMA3 and Gemma3 families, we demonstrate that this degradation is widespread across SLMs. In some settings, models trained on only 8k long CoT examples lose up to 75% of their original performance before fine-tuning. Strikingly, we further observe that for some particularly small models, even training on 220k long CoT examples fails to recover or surpass their original performance prior to fine-tuning. Our analysis attributes this effect to error accumulation: while longer responses increase the capacity for multi-step reasoning, they also amplify the risk of compounding mistakes. Furthermore, we find that Long CoT Degradation may negatively impacts downstream reinforcement learning (RL), although this can be alleviated by sufficiently scaled supervised fine-tuning (SFT). Our findings challenge common assumptions about the benefits of long CoT training for SLMs and offer practical guidance for building more effective small-scale reasoning models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。