根据硬件和批量大小动态调整树结构，显著加速大模型推理。
Inference-Cost-Aware Dynamic Tree Construction for Efficient Inference in Large Language Models
- 基于推理成本动态构建树结构，考虑GPU配置与批处理大小。
- 在6个任务上提速最高达5.2倍，比现有方法快5%~20%。
- 适合追求高效推理的工程师和部署者，尤其在多设备场景下。
大语言模型因自回归设计和模型规模庞大，面临严重的推理延迟问题。为解决此问题，推测解码成为有效方案，可同时生成并验证多个词元。尽管近期如EAGLE-2和EAGLE-3等方法通过动态树结构改进推测解码，但常忽略GPU设备、批量大小等关键系统变量的影响。为此，我们提出一种新动态树解码方法CAST，综合考虑推理成本（包括GPU配置与批量大小），动态优化树结构。在六种不同任务及六种不同LLM上的全面实验表明，该方法相比传统解码速度提升最高达5.2倍，普遍优于当前最先进技术5%至20%。代码已开源：https://github.com/EAGLE-Research/sglang-eagle4。
原文摘要 · Abstract (English)
Large Language Models (LLMs) face significant inference latency challenges stemming from their autoregressive design and large size. To address this, speculative decoding emerges as a solution, enabling the simultaneous generation and validation of multiple tokens. While recent approaches like EAGLE-2 and EAGLE-3 improve speculative decoding using dynamic tree structures, they often neglect the impact of crucial system variables such as GPU devices and batch sizes. Therefore, we introduce a new dynamic tree decoding approach called CAST that takes into account inference costs, including factors such as GPU configurations and batch sizes, to dynamically refine the tree structure. Through comprehensive experimentation across six diverse tasks and utilizing six distinct LLMs, our methodology demonstrates remarkable results, achieving speeds up to 5.2 times faster than conventional decoding methods. Moreover, it generally outperforms existing state-of-the-art techniques from 5 % to 20%. The code is available at https://github.com/EAGLE-Research/sglang-eagle4.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。