让模型从不完整的推理中学习,减少计算开销同时保持准确率。
Learning from Partial Chain-of-Thought via Truncated-Reasoning Self-Distillation
- 用冻结的教师模型生成完整推理,指导学生模型仅凭部分推理作答。
- 在多个基准上验证,截断推理时准确率损失小,且推理长度自然缩短。
- 适合追求高效推理、部署资源有限的场景,尤其适配大模型轻量化应用。
以推理为导向的语言模型通过生成长链式思维(chain-of-thought)在推理阶段表现优异,但带来显著且常过高的计算成本,导致冗余或低效推理。本文研究此问题,提出一种轻量级后训练方法——截断推理自蒸馏(Truncated-Reasoning Self-Distillation, TRSD)。TRSD中,一个固定教师模型首先生成完整推理轨迹,并基于提示和完整推理条件化生成答案分布,构建合成训练目标;随后,具有相同架构的学生模型被训练,仅依赖其推理轨迹的截断前缀来匹配教师的答案分布。在多个推理基准和不同词元预算下,实验表明TRSD能显著提升对截断推理的鲁棒性,且在多种推理模型上均实现较小的准确率损失。此外,尽管训练中未显式正则化短生成,我们发现TRSD训练后的模型在无截断情况下也自然输出更短的推理轨迹,从而在推理时大幅降低计算成本,无需人为干预。
原文摘要 · Abstract (English)
Reasoning-oriented language models achieve strong performance by generating long chain-of-thought traces at inference time. However, this capability comes with substantial and often excessive computational cost, which can materialize in redundant or inefficient reasoning. We study this setting and introduce Truncated-Reasoning Self-Distillation (TRSD), a lightweight post-training procedure that encourages models to produce correct predictions from partial reasoning traces. In TRSD, a frozen teacher model first generates a full reasoning trace and evaluates the corresponding answer distribution conditioned on the prompt and the complete reasoning to construct a synthetic training target. A student model with the same architecture is then trained to match the teacher's answer distribution while being conditioned only on a truncated prefix of its reasoning trace. Across multiple reasoning benchmarks and token budgets, we demonstrate that TRSD improves robustness to truncated inference, with far reduced accuracy tradeoffs when applied to a diverse set of reasoning models. Moreover, although never explicitly regularized for shorter generation during training, we also find that TRSD-trained models inherently output shorter reasoning traces without truncation, significantly reducing inference-time costs even without artificial interventions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。