提出SpecHub方法，提升多草稿推测解码的生成效率
SpecHub: Provable Acceleration to Multi-Draft Speculative Decoding
- 将最优传输问题简化为线性规划，降低计算复杂度
- 实验显示每步生成多0.05~0.27个标记，优于现有方法
- 适合追求推理速度的LLM部署场景
大语言模型在自然语言处理中至关重要，但其逐令牌生成限制了推理速度。多草稿推测解码（MDSD）通过小模型生成多个候选序列，由目标模型并行验证来加速。然而，现有启发式方法如递归拒绝采样（RRS）在后续草稿中接受率低，难以发挥多草稿优势。虽然基于成员成本的最优传输（OTM）理论上可提升接受率，但计算开销过大无法实时使用。本文提出SpecHub，一种高效采样-验证方法，在仅线性计算开销下提升接受率。通过将OTM问题简化为紧凑的线性规划模型，显著降低复杂度，并利用稀疏联合分布聚焦高概率序列，加速采样。大量实验表明，SpecHub每步生成0.05–0.27个额外标记，优于RRS；在无替换情况下，较RRS多生成0.02–0.16个标记。代码已开源。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have become essential in advancing natural language processing (NLP) tasks, but their sequential token generation limits inference speed. Multi-Draft Speculative Decoding (MDSD) offers a promising solution by using a smaller draft model to generate multiple token sequences, which the target LLM verifies in parallel. However, current heuristic approaches, such as Recursive Rejection Sampling (RRS), suffer from low acceptance rates in subsequent drafts, limiting the advantages of using multiple drafts. Meanwhile, Optimal Transport with Membership Cost (OTM) can theoretically improve acceptance rates, but its computational cost is too high for real-time use. We present SpecHub, a novel, efficient sampling-verification method for MDSD that improves acceptance rates with only linear computational overhead. By simplifying the OTM problem into a compact Linear Programming model, SpecHub significantly reduces computational complexity. It further accelerates sampling by leveraging a sparse joint distribution, focusing computation on high-probability token sequences. In extensive experiments, Spechub consistently generates 0.05-0.27 and 0.02-0.16 more tokens per step than RRS and RRS without replacement. We attach our code at \url{https://github.com/MasterGodzilla/Speculative_decoding_OT}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。