提出可证明无遗憾的草稿模型选择算法，显著提升大模型推理速度。
Not-a-Bandit: Provably No-Regret Drafter Selection in Speculative Decoding for LLMs
- 基于在线学习设计算法，无需额外调用主模型即可评估所有草稿模型。
- 在多草稿场景下性能随草稿数量指数级超越传统方法。
- 适用于各类推测解码框架，特别适合长链条推理任务。
推测解码广泛用于加速大语言模型（LLM）推理。本文聚焦于推测解码中的在线草稿模型选择问题，设计了一种算法，能够为每个查询在事后与最优草稿模型相比，实现令牌接受概率或期望接受长度上的无遗憾表现。关键创新在于无需额外调用目标模型即可准确评估所有草稿模型，从而在草稿模型数量增加时，性能相比现有基于强化学习的方法呈指数级提升。该方法适用于任意推测解码范式（单草稿、多草稿及草稿树）。我们还设计了系统高效的在线学习变体，显著降低计算与延迟开销。在开源LLM和多样数据集上进行的大量实验表明，本方法在多种领域中大幅超越当前最优的EAGLE3与BanditSpec基线，尤其在需要长推理链的任务中优势明显。
原文摘要 · Abstract (English)
Speculative decoding is widely used in accelerating large language model (LLM) inference. In this work, we focus on the online draft model selection problem in speculative decoding. We design an algorithm that provably competes with the best draft model in hindsight for each query in terms of either the token acceptance probability or expected acceptance length. In particular, we show that we can accurately evaluate all draft models, instead of only the chosen model without incurring additional queries to the target model, which allows us to improve exponentially over the existing bandit-based approach as the number of draft models increases. Our approach is generically applicable with any speculative decoding methods (single draft, multi-drafts and draft-trees). Moreover, we design system-efficient versions of online learners and demonstrate that the overhead in computation and latency can be substantially reduced. We conduct extensive experiments on open-source LLMs and diverse datasets, demonstrating that our methods substantially outperform the state-of-the-art EAGLE3 and the BanditSpec baseline in a variety of domains where specialized domain-expert drafters are available, especially when long reasoning chains are required.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。