KnapSpec动态选层加速大模型推理,适配长文本场景。
KnapSpec: Self-Speculative Decoding via Adaptive Layer Selection as a Knapsack Problem
- 将选层问题建模为背包问题,动态优化跳过层数
- 实测最高提升1.47倍推理速度,保持输出质量不变
- 无需训练,适配长序列,适合部署在真实硬件上
自推测解码(SSD)通过跳过部分网络层生成高效草稿模型以加速大语言模型推理,但现有方法多依赖静态启发式策略,忽视了长上下文场景下注意力计算的动态开销。本文提出KnapSpec,一种无需训练的框架,将草稿模型选择重构为背包问题,旨在最大化每秒生成词元数。通过分离注意力与MLP层,并将其硬件相关延迟建模为上下文长度的函数,KnapSpec利用并行动态规划算法实时识别最优草稿配置。此外,我们首次提供严谨理论分析,证明隐藏状态之间的余弦相似度是令牌接受率的数学合理代理。该基础使方法在实际硬件瓶颈变化时仍能保持高拟合度。在Qwen3和Llama3上的实验表明,KnapSpec持续优于当前最先进的SSD基线,在多个基准测试中实现最高1.47倍的墙钟时间加速。其即插即用特性确保长序列高速推理,无需额外训练且不改变目标模型输出分布。
原文摘要 · Abstract (English)
Self-speculative decoding (SSD) accelerates LLM inference by skipping layers to create an efficient draft model, yet existing methods often rely on static heuristics that ignore the dynamic computational overhead of attention in long-context scenarios. We propose KnapSpec, a training-free framework that reformulates draft model selection as a knapsack problem to maximize tokens-per-time throughput. By decoupling Attention and MLP layers and modeling their hardware-specific latencies as functions of context length, KnapSpec adaptively identifies optimal draft configurations on the fly via a parallel dynamic programming algorithm. Furthermore, we provide the first rigorous theoretical analysis establishing cosine similarity between hidden states as a mathematically sound proxy for the token acceptance rate. This foundation allows our method to maintain high drafting faithfulness while navigating the shifting bottlenecks of real-world hardware. Our experiments on Qwen3 and Llama3 demonstrate that KnapSpec consistently outperforms state-of-the-art SSD baselines, achieving up to 1.47x wall-clock speedup across various benchmarks. Our plug-and-play approach ensures high-speed inference for long sequences without requiring additional training or compromising the target model's output distribution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。