提升长文本推理速度，无需重训练即可实现2.84倍加速
SpecExtend: A Drop-in Enhancement for Speculative Decoding of Long Sequences
- 引入跨模型检索的缓存策略，动态优化长序列上下文选择
- 在16K-token文档摘要上提速2.84倍，长推理任务达3.86倍加速
- 兼容现有框架，不损失短文本性能，适合部署长序列生成场景
推测解码是加速大语言模型推理的常用技术，但其性能随输入长度增长显著下降，甚至在中等长度下已出现明显退化，这一问题长期未受关注。本文提出SpecExtend，一种无需额外训练的即插即用增强方案，通过集成FlashAttention与Hybrid Tree Attention，加速预填充和验证阶段。为在不重新训练的情况下提升长输入下的草案准确率与速度，提出跨模型检索（Cross-model Retrieval）机制，利用目标模型注意力分数动态选择适配小模型的上下文。大量实验表明，SpecExtend在16K token长文档摘要任务上实现最高2.84倍加速，在长形式推理任务上达3.86倍加速，同时保持现有先进框架在短输入上的性能表现。代码已开源。
原文摘要 · Abstract (English)
Speculative decoding is a widely used technique for accelerating inference in large language models (LLMs), but its performance degrades as input length grows, with significant drops even at moderate lengths. Yet, this early degradation has remained largely underexplored. We introduce SpecExtend, a drop-in enhancement that improves speculative decoding on long sequences without additional training. SpecExtend integrates efficient attention mechanisms such as FlashAttention and Hybrid Tree Attention to accelerate prefill and verification steps. To improve both draft accuracy and speed on long inputs without retraining, we propose Cross-model Retrieval, a novel KV cache eviction strategy that leverages the target model's attention scores to dynamically select relevant context for the smaller draft model. Extensive evaluations show that SpecExtend accelerates speculative decoding by up to 2.84x on 16K-token long document summarization and up to 3.86x on long-form reasoning, while preserving the short-input performance of state-of-the-art frameworks. Our code is available at https://github.com/jycha98/SpecExtend .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。