在昇腾芯片上实现稳定高效的树形推测解码，提升大模型推理速度。
EAGLE-Pangu: Accelerator-Safe Tree Speculative Decoding on Ascend NPUs
- 设计专用缓存管理器与安全张量化机制，适配昇腾架构
- 实测平均吞吐提升27%，尾延迟最高提升146%
- 支持可复现调试，适合部署优化与性能研究者
自回归解码仍是大语言模型服务的主要瓶颈，推测解码通过每步验证多个候选词减少昂贵的教师模型调用。树形推测进一步提升并行度，但在异构后端和加速器栈间常因注意力掩码、键值缓存布局和索引语义不兼容而失效。本文提出EAGLE-Pangu，将EAGLE-3风格的树形推测解码系统性地移植至昇腾NPU上的Pangu教师模型。该系统贡献：(i) 基于缓存API构建的显式分支/提交缓存管理器；(ii) 加速器安全的树形张量化，通过构造避免未定义负索引并验证结构不变性；(iii) 兼容融合内核的教师验证路径及可调试的急进回退。在MT-Bench和HumanEval类提示下240轮测试中，相对于仅使用教师模型的贪婪解码，其融合内核路径平均吞吐提升1.27倍，最差99%分位点最高提升2.46倍。同时提供无融合内核的参考路径，包含结构化追踪与不变性检查，支持跨执行模式和树预算的可复现调试与消融实验。
原文摘要 · Abstract (English)
Autoregressive decoding remains a primary bottleneck in large language model (LLM) serving, motivating speculative decoding methods that reduce expensive teacher-model invocations by verifying multiple candidate tokens per step. Tree-structured speculation further increases parallelism, but is often brittle when ported across heterogeneous backends and accelerator stacks, where attention masking, KV-cache layouts, and indexing semantics are not interchangeable. We present EAGLE-Pangu, a reproducible system that ports EAGLE-3-style tree speculative decoding to a Pangu teacher backend on Ascend NPUs. EAGLE-Pangu contributes (i) an explicit branch/commit cache manager built on the Cache API, (ii) accelerator-safe tree tensorization that removes undefined negative indices by construction and validates structural invariants, and (iii) a fused-kernel-compatible teacher verification path with a debuggable eager fallback. On 240 turns from MT-Bench and HumanEval-style prompts, EAGLE-Pangu improves end-to-end decoding throughput by 1.27x on average, up to 2.46x at p99, over teacher-only greedy decoding in the fused-kernel performance path. We also provide a fused-kernel-free reference path with structured traces and invariant checks to support reproducible debugging and ablation across execution modes and tree budgets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。