动态深度解码让大模型推理快近3倍，且不损失准确率。
Dynamic Depth Decoding: Faster Speculative Decoding for LLMs
- 根据生成进度动态调整预测深度，优化树形草稿策略。
- 相比EAGLE-2提升44%速度，平均加速达3.16倍。
- 适合追求高吞吐、低延迟的大模型部署场景。
使用推测解码加速大语言模型可显著提升运行效率，且不损失准确性。当前最优方法EAGLE-2通过动态草稿树实现性能突破。本文提出动态深度解码（DDD），通过引入动态深度机制优化EAGLE-2的树形草稿方法，使平均速度比EAGLE-2进一步提升44%，最终达到3.16倍的平均加速效果。该方法在多个基准测试中保持与原始模型一致的输出质量。
原文摘要 · Abstract (English)
The acceleration of Large Language Models (LLMs) with speculative decoding provides a significant runtime improvement without any loss of accuracy. Currently, EAGLE-2 is the state-of-the-art speculative decoding method, improving on EAGLE with a dynamic draft tree. We introduce Dynamic Depth Decoding (DDD), which optimises EAGLE-2's tree drafting method using a dynamic depth. This extends the average speedup that EAGLE-2 achieves over EAGLE by $44\%$, giving DDD an average speedup of $3.16$x.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。