通过显式建模动态生成结构，提升大模型推理速度。
AdaEAGLE: Optimizing Speculative Decoding via Explicit Modeling of Adaptive Draft Structures
- 用轻量级预测模块动态确定生成令牌数。
- 相比传统方法提速1.62倍，保持输出质量。
- 无需人工调参，适合追求高效推理的场景。
推测解码（Speculative Decoding, SD）是加速大语言模型（LLMs）推理的热门无损技术。本文发现，采用上下文感知的自适应生成结构可显著提升静态生成结构的解码速度。然而，现有自适应结构研究受限于性能、建模方式与适用性。为此，我们提出AdaEAGLE，首个显式建模自适应生成结构的SD框架。该框架利用轻量级生成长度预测器（LDLP）模块，在推理时显式预测最优生成令牌数以指导生成模型。它无需人工设定阈值，即可实现接近最先进水平的速度提升，并支持更深层次的优化。结合阈值策略，AdaEAGLE在保持输出质量的前提下，相较原始自回归解码实现1.62倍加速，优于固定长度的最先进基线。
原文摘要 · Abstract (English)
Speculative Decoding (SD) is a popular lossless technique for accelerating the inference of Large Language Models (LLMs). We show that the decoding speed of SD frameworks with static draft structures can be significantly improved by incorporating context-aware adaptive draft structures. However, current studies on adaptive draft structures are limited by their performance, modeling approaches, and applicability. In this paper, we introduce AdaEAGLE, the first SD framework that explicitly models adaptive draft structures. AdaEAGLE leverages the Lightweight Draft Length Predictor (LDLP) module to explicitly predict the optimal number of draft tokens during inference to guide the draft model. It achieves comparable speedup results without manual thresholds and allows for deeper, more specialized optimizations. Moreover, together with threshold-based strategies, AdaEAGLE achieves a $1.62\times$ speedup over the vanilla AR decoding and outperforms fixed-length SotA baseline while maintaining output quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。