用小模型快速起草思考过程,大幅提升大模型推理速度。
Efficient Reasoning for LLMs through Speculative Chain-of-Thought
- 小模型先草拟思考链,大模型只修正错误部分。
- 在多个数学数据集上降低48%~66%延迟,性能接近原模型。
- 适合追求高速推理又不想牺牲精度的AI应用开发者。
大型推理语言模型如OpenAI-o1和Deepseek-R1因其强大的任务解决能力受到广泛关注。然而,巨大的模型规模和长思考链导致显著的推理成本与响应延迟。现有高效推理方法主要集中在减少参数量或缩短思维链长度。本文提出推测性思维链(SCoT),通过大、小模型协作从另一角度加速平均推理速度。SCoT利用轻量级草稿模型进行思维层级的预演,随后选择最优思维链草案,并由目标模型修正错误案例。所提出的思维行为对齐机制提升了草稿效率,草案选择策略保持了目标模型在复杂任务上的预测准确性。在GSM8K、MATH、GaoKao、CollegeMath和Olympiad数据集上的实验结果表明,SCoT使Deepseek-R1-Distill-Qwen-32B和Deepseek-R1-Distill-Llama-70B的推理延迟分别降低了48%~66%和21%~49%,同时达到接近目标模型水平的性能。代码已公开于https://github.com/Jikai0Wang/Speculative_CoT。
原文摘要 · Abstract (English)
Large reasoning language models such as OpenAI-o1 and Deepseek-R1 have recently attracted widespread attention due to their impressive task-solving abilities. However, the enormous model size and the generation of lengthy thought chains introduce significant reasoning costs and response latency. Existing methods for efficient reasoning mainly focus on reducing the number of model parameters or shortening the chain-of-thought length. In this paper, we introduce Speculative Chain-of-Thought (SCoT), which reduces reasoning latency from another perspective by accelerated average reasoning speed through large and small model collaboration. SCoT conducts thought-level drafting using a lightweight draft model. Then it selects the best CoT draft and corrects the error cases with the target model. The proposed thinking behavior alignment improves the efficiency of drafting and the draft selection strategy maintains the prediction accuracy of the target model for complex tasks. Experimental results on GSM8K, MATH, GaoKao, CollegeMath and Olympiad datasets show that SCoT reduces reasoning latency by 48\%$\sim$66\% and 21\%$\sim$49\% for Deepseek-R1-Distill-Qwen-32B and Deepseek-R1-Distill-Llama-70B while achieving near-target-model-level performance. Our code is available at https://github.com/Jikai0Wang/Speculative_CoT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。