AdaSD让大模型推理更高效，无需调参即可自动优化生成速度和准确率。
AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference
- 动态调整生成长度与采纳标准，基于熵和距离实时决策。
- 相比原版推测解码最高提速46%，准确率损失低于1.8%。
- 无需预分析或微调，直接适配现成模型，实用性强。
大语言模型在众多任务中表现卓越，但参数规模增大导致推理速度下降。推测解码通过小型草稿模型预测候选词，再由大型目标模型验证来缓解此问题。然而，现有方法通常需额外训练、大量超参数调优或部署前的模型与任务分析。本文提出自适应推测解码（AdaSD），一种无需超参数的解码方案，可在推理过程中动态调整生成长度和接受标准。AdaSD引入两个自适应组件：一个决定何时停止候选词生成，另一个决定词的接受与否，均基于词元熵和Jensen-Shannon距离实时更新。该方法免去预分析或微调，兼容现成模型。在基准数据集上的实验表明，AdaSD相较原始推测解码最高实现1.46倍加速，且准确率下降不超过1.8%，为高效、自适应的大语言模型推理提供了实用解决方案。
原文摘要 · Abstract (English)
Large language models (LLMs) have achieved remarkable performance across a wide range of tasks, but their increasing parameter sizes significantly slow down inference. Speculative decoding mitigates this issue by leveraging a smaller draft model to predict candidate tokens, which are then verified by a larger target model. However, existing approaches often require additional training, extensive hyperparameter tuning, or prior analysis of models and tasks before deployment. In this paper, we propose Adaptive Speculative Decoding (AdaSD), a hyperparameter-free decoding scheme that dynamically adjusts generation length and acceptance criteria during inference. AdaSD introduces two adaptive components: one to determine when to stop candidate token generation and the other to decide token acceptance, updated in real time based on token entropy and Jensen-Shannon distance. This approach eliminates the need for pre-analysis or fine-tuning and is compatible with off-the-shelf models. Experiments on benchmark datasets demonstrate that AdaSD achieves up to 1.46x speedup over vanilla speculative decoding while limiting accuracy degradation to under 1.8%, making it a practical solution for efficient and adaptive LLM inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。