解决长文本生成中推测解码速度下降问题，提升5倍接受长度。
OWL: Overcoming Window Length-Dependence in Speculative Decoding for Long-Context Inputs
- 用仅依赖最后词元状态的LSTM小模型作草稿，适配任意长度上下文
- 在验证器中引入[SPEC]特殊标记，增强草稿匹配能力，长文本下接受长度提升5倍
- 融合树形与非树形解码策略，兼顾效率与稳定性，适合实际长文本场景
推测解码可加速大语言模型推理，但现有方法难以适应真实场景。基准测试通常假设短上下文（如2K token），而实际应用涉及长上下文。我们发现当前方法在长上下文下性能严重退化，例如EAGLE3反而使生成速度降低至0.81x。为此，我们发布新的长上下文基准（LongSpecBench），并提出新模型OWL。OWL通过三项创新，在长上下文输入下实现约5倍于EAGLE3的接受长度：（1）采用仅基于最后一词元状态的LSTM草稿模型，具备跨长度泛化能力；（2）在验证器中引入特殊标记[SPEC]，生成更丰富的表示以提升草稿匹配效果；（3）结合树形与非树形解码的混合算法。所有代码与数据集均已开源，推动后续研究。
原文摘要 · Abstract (English)
Speculative decoding promises faster inference for large language models (LLMs), yet existing methods fail to generalize to real-world settings. Benchmarks typically assume short contexts (e.g., 2K tokens), whereas practical workloads involve long contexts. We find current approaches degrade severely with long contexts; for instance, EAGLE3 even slows down the generation speed by 0.81x. We address these limitations by releasing a new long-context benchmark (LongSpecBench) and introducing a novel model (OWL). OWL achieves about 5x higher acceptance length than EAGLE3 on long-context inputs through three innovations: (1) an LSTM-based drafter conditioned only on the last-token state, making it generalize to various lengths, (2) a special token [SPEC] in the verifier that produces richer representation for drafter, and (3) a hybrid algorithm combining both tree and non-tree decoding methods. We release all code and datasets to advance future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。