不训练、轻量级预测关键令牌，显著加速大模型首次输出速度。
Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimation
- 用轻量模型预测提示中重要令牌，只传关键部分给主模型
- 实测使Llama-3.1-405B的端到端吞吐提升7倍，首字延迟改善7.66倍
- 适合对响应速度敏感的长文本生成场景，无需额外训练
提升首次输出时间（TTFT）是现代大语言模型推理引擎的核心目标。优化TTFT直接提升最大每秒查询数（QPS），满足诸多关键应用需求。然而，由于计算受限，提升TTFT极具挑战性，性能瓶颈已从以往关注的自注意力层转移至MLP部分。本文提出SpecPrefill，一种无需训练的框架，通过以下洞察加速长、中等上下文查询的推理TTFT：大模型具备泛化能力，仅需精心选取的提示令牌子集即可保持质量。核心思想是利用轻量模型基于上下文推测局部重要令牌，连同必要位置信息一并送入主模型处理。在多样任务上评估，并进行端到端与消融实验。结果表明，SpecPrefill在真实下游任务中实现对Llama-3.1-405B-Instruct-FP8高达7倍的端到端最大QPS提升，且TTFT改善达7.66倍。
原文摘要 · Abstract (English)
Improving time-to-first-token (TTFT) is an essentially important objective in modern large language model (LLM) inference engines. Optimizing TTFT directly results in higher maximal QPS and meets the requirements of many critical applications. However, boosting TTFT is notoriously challenging since it is compute-bounded and the performance bottleneck shifts from the self-attention that many prior works focus on to the MLP part. In this work, we present SpecPrefill, a training free framework that accelerates the inference TTFT for both long and medium context queries based on the following insight: LLMs are generalized enough to preserve the quality given only a carefully chosen subset of prompt tokens. At its core, SpecPrefill leverages a lightweight model to speculate locally important tokens based on the context. These tokens, along with the necessary positional information, are then sent to the main model for processing. We evaluate SpecPrefill with a diverse set of tasks, followed by a comprehensive benchmarking of performance improvement both in a real end-to-end setting and ablation studies. SpecPrefill manages to serve Llama-3.1-405B-Instruct-FP8 with up to 7$\times$ maximal end-to-end QPS on real downstream tasks and 7.66$\times$ TTFT improvement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。