Dustin通过稀疏验证提升长文本生成效率，显著降低推理延迟。
Dustin: Draft-Augmented Sparse Verification for Efficient Long-Context Generation with Speculative Decoding
- 结合草稿模型与目标模型历史注意力，精准识别关键词元。
- 在32k序列长度下实现9.17倍端到端速度提升，误差可忽略。
- 适合需要高效长文本生成的部署场景，如大模型推理服务。
尽管推测解码能提升多批次长上下文大语言模型的推理吞吐量，但其效率常受限于验证瓶颈——键值缓存加载主导延迟。现有压缩方法在此场景失效：静态淘汰导致因显著性偏移引发准确率下降，动态选择则在验证路径引入过高计算开销。本文提出Dustin，一种专为长上下文推测解码设计的稀疏验证框架。Dustin融合草稿模型的前瞻信号与目标模型的历史注意力，以高保真度识别多步验证窗口中的关键词元。为进一步降低重计算延迟，该方法采用稀疏估计策略，仅对最小数量的注意力头进行重要性评分。在PG-19和LongBench数据集上使用Qwen2.5-72B模型评估显示，Dustin在32k序列长度下实现自注意力加速27.85倍，端到端解码速度提升9.17倍，且准确率几乎无损。
原文摘要 · Abstract (English)
While speculative decoding improves inference throughput for multi-batch long-context Large Language Models (LLMs), its efficiency is often limited by a verification bottleneck where Key-Value (KV) cache loading dominates latency. Existing compression methods fail in this regime: static eviction incurs accuracy loss due to saliency shift, while dynamic selection introduces prohibitive computational overhead during the verification path. We propose Dustin, a sparse verification framework designed for long-context speculative decoding. Dustin integrates lookahead signals from the draft model with historical attention from the target model to identify critical tokens with high fidelity across multi-step verification windows. To reduce recomputation latency, this approach further employs a sparse estimation scheme that restricts importance scoring to a minimal subset of attention heads. Evaluations on PG-19 and LongBench with Qwen2.5-72B demonstrate that Dustin achieves a 27.85x speedup in self-attention and a 9.17x end-to-end decoding speedup at a 32k sequence length, all with negligible accuracy degradation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。