无需训练即可提升生成效率与准确率的推测解码方法。
Alignment-Augmented Speculative Decoding with Alignment Sampling and Conditional Verification
- 利用预填充阶段输出分布生成更对齐的草稿序列。
- 通过自适应阈值验证，提升生成准确率3.3分，速度提升2.23倍。
- 适合追求高效推理且无训练资源的场景，如部署优化。
近期研究表明，推测解码在加速大语言模型自回归生成方面具有巨大潜力。这些方法的成功依赖于草稿候选与目标模型采样输出之间的对齐性。现有方法主要通过训练方式实现对齐（如EAGLE、Medusa），但代价高昂。本文提出一种无需训练的对齐增强型推测解码算法。我们设计了对齐采样机制，利用预填充阶段获得的输出分布生成更对齐的草稿候选。为进一步利用高质量但非对齐的草稿，引入简单有效的灵活验证策略，通过自适应概率阈值提升生成准确性并进一步提高推理效率。在8个数据集（涵盖问答、摘要、代码补全任务）上的实验表明，该方法使LLaMA3模型平均生成得分提升3.3点，平均接受长度达2.39，推理速度提升2.23倍。
原文摘要 · Abstract (English)
Recent works have revealed the great potential of speculative decoding in accelerating the autoregressive generation process of large language models. The success of these methods relies on the alignment between draft candidates and the sampled outputs of the target model. Existing methods mainly achieve draft-target alignment with training-based methods, e.g., EAGLE, Medusa, involving considerable training costs. In this paper, we present a training-free alignment-augmented speculative decoding algorithm. We propose alignment sampling, which leverages output distribution obtained in the prefilling phase to provide more aligned draft candidates. To further benefit from high-quality but non-aligned draft candidates, we also introduce a simple yet effective flexible verification strategy. Through an adaptive probability threshold, our approach can improve generation accuracy while further improving inference efficiency. Experiments on 8 datasets (including question answering, summarization and code completion tasks) show that our approach increases the average generation score by 3.3 points for the LLaMA3 model. Our method achieves a mean acceptance length up to 2.39 and speed up generation by 2.23.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。