提出分层推测解码，突破联合不可行性，显著提升生成效率。
Overcoming Joint Intractability with Lossless Hierarchical Speculative Decoding
- 采用分层结构平衡各路径概率质量，实现无损验证。
- 在多模型、多基准上提升接受率，集成EAGLE-3后性能超12%。
- 方法可解释性强，适用于各类推测解码框架，适合高效推理研究者。
验证是提升推测解码推理速度并保持分布一致性的关键瓶颈。现有研究表明，序列级验证相比逐标记验证能获得更高的通过令牌数。然而，现有方法常依赖近似替代或受限于部分信息，难以解决联合不可行性问题。本文提出分层推测解码（HSD），一种可证明无损的验证方法，通过在可访问分支间平衡过量与不足的概率质量，显著提升预期通过令牌数，并克服联合不可行性。大规模实验表明，HSD在多种模型家族和基准测试中均实现稳定的接受率提升。其强可解释性与通用性使其可无缝集成到广泛推测解码框架中。值得注意的是，将其集成至EAGLE-3后，性能提升超过12%，在不牺牲分布保真度的前提下达到当前最优解码效率。代码已开源：https://github.com/ZhouYuxuanYX/Hierarchical-Speculative-Decoding。
原文摘要 · Abstract (English)
Verification is a key bottleneck in improving inference speed while maintaining distribution fidelity in Speculative Decoding. Recent work has shown that sequence-level verification leads to a higher number of accepted tokens compared to token-wise verification. However, existing solutions often rely on surrogate approximations or are constrained by partial information, struggling with joint intractability. In this work, we propose Hierarchical Speculative Decoding (HSD), a provably lossless verification method that significantly boosts the expected number of accepted tokens and overcomes joint intractability by balancing excess and deficient probability mass across accessible branches. Our extensive large-scale experiments demonstrate that HSD yields consistent improvements in acceptance rates across diverse model families and benchmarks. Moreover, its strong explainability and generality make it readily integrable into a wide range of speculative decoding frameworks. Notably, integrating HSD into EAGLE-3 yields over a 12% performance gain, establishing state-of-the-art decoding efficiency without compromising distribution fidelity. Code is available at https://github.com/ZhouYuxuanYX/Hierarchical-Speculative-Decoding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。