基于时间局部性的分层草稿机制，无损提升大模型推理速度。
Lossless Acceleration of Large Language Models with Hierarchical Drafting based on Temporal Locality in Speculative Decoding
- 按时间局部性分层管理词元来源，逐步检索更高置信草稿。
- 7B/13B模型在多任务下均实现稳定加速，最高提速2.5倍。
- 无需微调，适用于不同温度与任务，适合实时交互场景。
大语言模型（LLM）的推理加速对实时交互至关重要，已广泛应用于实际服务中。推测解码作为一种全算法解决方案，通过草稿和验证词元，在单次前向传播中生成多个词元，从而提升推理速度。然而，当前草稿策略通常需要大量微调，或在不同任务间表现不一。为此，我们提出分层草稿（Hierarchy Drafting, HD），一种新型无损草稿方法，基于时间局部性将多种词元来源组织成多层次数据库。在草稿阶段，HD按从高到低局部性顺序逐层访问数据库，获取草稿词元，确保跨任务的一致加速并最小化草稿延迟。在包含7B和13B参数的LLM上进行的Spec-Bench实验表明，HD优于现有数据库草稿方法，在不同模型规模、任务和温度条件下均表现出稳健的推理加速效果。
原文摘要 · Abstract (English)
Accelerating inference in Large Language Models (LLMs) is critical for real-time interactions, as they have been widely incorporated into real-world services. Speculative decoding, a fully algorithmic solution, has gained attention for improving inference speed by drafting and verifying tokens, thereby generating multiple tokens in a single forward pass. However, current drafting strategies usually require significant fine-tuning or have inconsistent performance across tasks. To address these challenges, we propose Hierarchy Drafting (HD), a novel lossless drafting approach that organizes various token sources into multiple databases in a hierarchical framework based on temporal locality. In the drafting step, HD sequentially accesses multiple databases to obtain draft tokens from the highest to the lowest locality, ensuring consistent acceleration across diverse tasks and minimizing drafting latency. Our experiments on Spec-Bench using LLMs with 7B and 13B parameters demonstrate that HD outperforms existing database drafting methods, achieving robust inference speedups across model sizes, tasks, and temperatures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。