用小模型辅助大模型，提速2.5倍且减少调用次数
Consultant Decoding: Yet Another Synergistic Mechanism
- 用大模型自身计算的词元概率验证候选生成，无需额外验证机制
- 推理速度提升2.5倍，大模型调用频率低于10%
- 跨量级模型协同，适合资源受限场景下的高效推理
基于推测解码（Speculative Decoding, SD）的协同机制因简单高效而受到广泛关注，但高拒绝率导致需多次调用大语言模型（LLM）验证生成词元，削弱了整体加速效果。本文重新审视现有验证机制，提出新型协同机制——顾问解码（Consultant Decoding, CD）。与依赖重要性采样度量的SD不同，CD仅使用大模型自身计算的词元级似然值进行候选验证。实验表明，CD相比目标模型实现最高达2.5倍的推理加速，生成质量维持在目标模型的约100%水平；该效果通过参数量相差两个数量级的模型协同实现，且大模型调用频率降至10%以下，尤其在高难度任务中表现更优。值得注意的是，其性能甚至超越了理论上作为推测解码上限的大模型本身。
原文摘要 · Abstract (English)
The synergistic mechanism based on Speculative Decoding (SD) has garnered considerable attention as a simple yet effective approach for accelerating the inference of large language models (LLMs). Nonetheless, the high rejection rates require repeated LLMs calls to validate draft tokens, undermining the overall efficiency gain of SD. In this work, we revisit existing verification mechanisms and propose a novel synergetic mechanism Consultant Decoding (CD). Unlike SD, which relies on a metric derived from importance sampling for verification, CD verifies candidate drafts using token-level likelihoods computed solely by the LLM. CD achieves up to a 2.5-fold increase in inference speed compared to the target model, while maintaining comparable generation quality (around 100% of the target model's performance). Interestingly, this is achieved by combining models whose parameter sizes differ by two orders of magnitude. In addition, CD reduces the call frequency of the large target model to below 10%, particularly in more demanding tasks. CD's performance was even found to surpass that of the large target model, which theoretically represents the upper bound for speculative decoding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。