用跨注意力设计新架构，让大模型推理更快更稳。
Cross-Attention Speculative Decoding
- 用跨注意力替代自注意力，简化结构不依赖额外组件。
- 在多个模型上实现与顶尖方法相当的加速效果。
- 适合追求高效训练与稳定推理的模型部署场景。
推测解码（Speculative Decoding, SD）是加速大语言模型推理的常用方法，尤其在草稿模型与目标模型对齐时表现优异。然而，当前最先进的SD方法通常依赖紧密耦合的自注意力Transformer解码器，并常辅以池化或融合层，导致架构复杂且难以泛化。本文提出预算型EAGLE（Beagle），据我们所知首个基于跨注意力的Transformer解码器SD模型，性能与领先自注意力模型EAGLE-v2相当，同时无需池化或辅助组件，简化了架构，提升了训练效率，并在训练仿真中保持稳定的内存使用。为支持该新型架构的有效训练，我们提出两阶段块注意力训练法（Two-Stage Block-Attention Training），在块级注意力场景下实现训练稳定与收敛高效。大量实验覆盖多个LLM与数据集表明，Beagle在推理加速方面表现竞争力，且训练效率优于EAGLE-v2，为推测解码架构提供了一种强有力的替代方案。
原文摘要 · Abstract (English)
Speculative decoding (SD) is a widely adopted approach for accelerating inference in large language models (LLMs), particularly when the draft and target models are well aligned. However, state-of-the-art SD methods typically rely on tightly coupled, self-attention-based Transformer decoders, often augmented with auxiliary pooling or fusion layers. This coupling makes them increasingly complex and harder to generalize across different models. We present Budget EAGLE (Beagle), the first, to our knowledge, cross-attention-based Transformer decoder SD model that achieves performance on par with leading self-attention SD models (EAGLE-v2) while eliminating the need for pooling or auxiliary components, simplifying the architecture, improving training efficiency, and maintaining stable memory usage during training-time simulation. To enable effective training of this novel architecture, we propose Two-Stage Block-Attention Training, a new method that achieves training stability and convergence efficiency in block-level attention scenarios. Extensive experiments across multiple LLMs and datasets show that Beagle achieves competitive inference speedups and higher training efficiency than EAGLE-v2, offering a strong alternative for architectures in speculative decoding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。