让小模型学会大模型的推理,关键在生成适配它的思考过程。
In Their Own Words: Reasoning Traces Tailored for Small Models Make Them Better Reasoners
- 用学生模型决定是否采纳教师提出的思路,过滤低概率词。
- 小模型用新方法训练后,推理能力提升4.9%,而原方法下降20.5%。
- 适合需要高效推理的小型模型部署场景,如移动端或边缘设备。
通过监督微调将大模型的推理能力迁移到小模型,常出现反直觉的性能下降,即使有高质量教师示范也如此。我们发现根源在于分布错位:大模型的推理轨迹包含学生模型低概率词汇,超出其内部表征容量,形成学习障碍而非帮助。为此提出逆向推测解码(RSD)机制,教师提出候选词,学生根据自身概率分布决定是否采纳,过滤低概率项。对Qwen3-0.6B实验显示,直接蒸馏s1K-1.1数据使基准测试平均性能下降20.5%,而使用RSD生成的轨迹训练后,性能反而提升4.9%。分析表明,低概率词是迁移的关键瓶颈。跨模型实验进一步证明,RSD轨迹具有模型特异性,需为每个学生架构定制分布对齐。
原文摘要 · Abstract (English)
Transferring reasoning capabilities from larger language models to smaller ones through supervised fine-tuning often fails counterintuitively, with performance degrading despite access to high-quality teacher demonstrations. We identify that this failure stems from distributional misalignment: reasoning traces from larger models contain tokens that are low probability under the student's distribution, exceeding the internal representation capacity of smaller architectures and creating learning barriers rather than helpful guidance. We propose Reverse Speculative Decoding (RSD), a mechanism for generating student-friendly reasoning traces in which the teacher model proposes candidate tokens but the student model determines acceptance based on its own probability distributions, filtering low probability tokens. When applied to Qwen3-0.6B, direct distillation of s1K-1.1 reasoning trace data degrades average performance across major reasoning benchmarks by 20.5\%, while the same model trained on RSD-generated reasoning traces achieves meaningful improvements of 4.9\%. Our analysis reveals that low probability tokens constitute the critical bottleneck in reasoning ability transfer. However, cross-model experiments demonstrate that RSD traces are model-specific rather than universally applicable, indicating that distributional alignment must be tailored for each student architecture's unique internal representation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。