用轻量代理减少模型调用，让大模型推理快35%。
TriSpec: Ternary Speculative Decoding via Lightweight Proxy Verification
- 引入轻量代理，只在不确定时调用完整模型
- 相比标准推测解码提速35%，调用次数少50%
- 适合追求高效推理的部署场景
大语言模型的推理效率受限于其串行自回归生成机制，尤其在推理任务变长时更为明显。推测解码（SD）通过轻量草稿与并行验证机制实现显著加速。现有工作已接近草稿效率的极限，本文从验证成本角度切入，提出新型三元推测解码框架TriSpec：核心是引入轻量代理，对易验证的草稿序列直接放行，仅在遇到不确定词元时才调用完整目标模型。TriSpec可与EAGLE-3等先进方法结合，进一步降低验证开销。在Qwen3、DeepSeek-R1-Distill-Qwen和LLaMA系列上的实验表明，相比标准SD，TriSpec最高可实现35%的加速，目标模型调用次数减少高达50%，且保持相当的准确率。
原文摘要 · Abstract (English)
Inference efficiency in Large Language Models (LLMs) is fundamentally limited by their serial, autoregressive generation, especially as reasoning becomes a key capability and response sequences grow longer. Speculative decoding (SD) offers a powerful solution, providing significant speed-ups through its lightweight drafting and parallel verification mechanism. While existing work has nearly saturated improvements in draft effectiveness and efficiency, this paper advances SD from a new yet critical perspective: the verification cost. We propose TriSpec, a novel ternary SD framework that, at its core, introduces a lightweight proxy to significantly reduce computational cost by approving easily verifiable draft sequences and engaging the full target model only when encountering uncertain tokens. TriSpec can be integrated with state-of-the-art SD methods like EAGLE-3 to further reduce verification costs, achieving greater acceleration. Extensive experiments on the Qwen3 and DeepSeek-R1-Distill-Qwen/LLaMA families show that TriSpec achieves up to 35\% speedup over standard SD, with up to 50\% fewer target model invocations while maintaining comparable accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。