Yggdrasil让大模型推理更快，通过智能预判和高效调度。
Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based LLM Decoding
- 设计等长树结构，让预判生成更符合运行时需求。
- 在多种硬件上实现最高3.98倍的推理加速。
- 无需修改大模型，适合追求低延迟的部署场景。
推测解码通过并行生成和验证多个标记提升大模型推理效率，但现有系统因动态推测与静态运行时假设不匹配而性能不佳。我们提出Yggdrasil，一种协同设计的系统，通过上下文感知的树状草稿和编译友好执行实现延迟最优的推测解码。Yggdrasil引入等长树结构以适配静态图，采用延迟感知优化目标选择草稿，并通过分阶段调度降低开销。该系统支持未经修改的大模型，在多种硬件配置下相比最先进基线最高实现3.98倍加速。
原文摘要 · Abstract (English)
Speculative decoding improves LLM inference by generating and verifying multiple tokens in parallel, but existing systems suffer from suboptimal performance due to a mismatch between dynamic speculation and static runtime assumptions. We present Yggdrasil, a co-designed system that enables latency-optimal speculative decoding through context-aware tree drafting and compiler-friendly execution. Yggdrasil introduces an equal-growth tree structure for static graph compatibility, a latency-aware optimization objective for draft selection, and stage-based scheduling to reduce overhead. Yggdrasil supports unmodified LLMs and achieves up to $3.98\times$ speedup over state-of-the-art baselines across multiple hardware setups.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。