用轻量模型直接预测退出信号，让大模型推理快2.5倍且不丢精度
SpecExit: Accelerating Large Reasoning Model via Speculative Exit
- 用轻量模型从隐藏状态直接预测下一步词和退出信号，无需额外探测开销
- 平均生成长度减少66%，端到端延迟降低2.5倍，准确率不变
- 适合追求高效推理的部署场景，尤其适合资源受限环境
尽管大型推理模型在推理任务上表现优异，但常因过度思考导致输出过长，造成显著的端到端延迟，限制了实际部署。为缓解该问题，早期退出机制被提出，可在常规完成前终止推理，有效缩短生成长度且对准确率影响极小。然而，其依赖探测机制引入了检测开销，限制了端到端延迟提升，并降低了在多样化问题上的通用性。受推测解码中隐藏状态应用的启发，我们提出SpecExit：一种新框架，通过轻量草稿模型直接从隐藏状态预测未来词与早期退出信号，无需探测开销。实验显示，该方法将平均生成长度减少66%，相比推测解码基线实现2.5倍端到端加速，且保持准确率不变。结果表明，隐藏状态蕴含有效退出信号，可广泛用于高效推理。代码已开源：https://github.com/Tencent/AngelSlim。
原文摘要 · Abstract (English)
Despite their strong performance on reasoning tasks, large reasoning models (LRMs) often suffer from overthinking, producing unnecessarily long outputs and incurring high end-to-end latency, a significant limitation to their real-world deployment. To address overthinking, early-exit mechanisms have been proposed to terminate reasoning before typical completion, showing that this approach can effectively shorten generation length with minimal impact on accuracy. However, their reliance on probing mechanisms introduces a detection overhead that limits their end-to-end latency gains and compromises their generalizability across diverse problems. Inspired by the use of hidden states in speculative decoding, we propose SpecExit, a novel framework that predicts both future tokens and an early-exit signal directly from a lightweight draft model without probing overhead. Our method offers significant improvements, reducing average generation length by 66\% and achieving a 2.5x speedup in end-to-end latency compared to the speculative decoding baseline, without compromising accuracy. Our method leverages the inherent signals from hidden states to provide effective early-exit signals, suggesting broader use of hidden states for efficient reasoning. Our code is available at https://github.com/Tencent/AngelSlim.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。