AdaSpec动态调整生成策略，让大模型推理更快更稳。
AdaSpec: Adaptive Speculative Decoding for Fast, SLO-Aware Large Language Model Serving
- 根据实时负载自动优化猜测生成方案
- 在真实服务数据上实现最高66%加速且达标SLO
- 适合需要低延迟和高可靠性的大模型部署场景
基于云的大语言模型服务常面临推理延迟高、难以满足服务等级目标（SLO）的问题，尤其在请求模式动态变化时。推测性解码通过轻量模型生成草案、大模型验证结果来加速推理，但现有方法难以适应波动负载与动态环境，导致性能下降和SLO违规。本文提出AdaSpec，一种可动态调整推测策略的高效推理系统，基于理论模型分析并预测不同场景下的策略效率，并实现智能草案生成与验证算法，在保证高SLO达成的同时最大化性能。在真实世界大模型服务轨迹上的实验表明，AdaSpec始终满足SLO要求，相比当前最优推测推理系统实现最高达66%的加速。代码已公开于https://github.com/cerebellumking/AdaSpec。
原文摘要 · Abstract (English)
Cloud-based Large Language Model (LLM) services often face challenges in achieving low inference latency and meeting Service Level Objectives (SLOs) under dynamic request patterns. Speculative decoding, which exploits lightweight models for drafting and LLMs for verification, has emerged as a compelling technique to accelerate LLM inference. However, existing speculative decoding solutions often fail to adapt to fluctuating workloads and dynamic system environments, resulting in impaired performance and SLO violations. In this paper, we introduce AdaSpec, an efficient LLM inference system that dynamically adjusts speculative strategies according to real-time request loads and system configurations. AdaSpec proposes a theoretical model to analyze and predict the efficiency of speculative strategies across diverse scenarios. Additionally, it implements intelligent drafting and verification algorithms to maximize performance while ensuring high SLO attainment. Experimental results on real-world LLM service traces demonstrate that AdaSpec consistently meets SLOs and achieves substantial performance improvements, delivering up to 66% speedup compared to state-of-the-art speculative inference systems. The source code is publicly available at https://github.com/cerebellumking/AdaSpec
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。