优化大模型推理，让Llama在生产环境提速10%以上
Efficient Speculative Decoding for Llama at Scale: Challenges and Solutions
- 用EAGLE框架实现树状注意力和多轮推测解码的高效GPU计算
- 单批次下每令牌解码仅需4毫秒，比现有方法快10%
- 支持大规模并发，吞吐提升1.4至2.0倍，适合工业级部署
推测解码是加速大语言模型推理的常用方法。然而，在生产环境中规模化应用面临诸多工程挑战，如在GPU上高效实现树状注意力与多轮推测解码等操作。本文详述了我们为实现基于EAGLE的推测解码在Llama模型上的生产级部署所采用的训练与推理优化技术。通过这些改进，我们在Llama模型上实现了新的推理延迟最优表现。例如，使用8块NVIDIA H100 GPU时，Llama4 Maverick在单批处理下的解码速度达到约每令牌4毫秒，较此前最佳方法提升10%。此外，对于基于EAGLE的推测解码，我们的优化在大规模批量处理下实现了1.4倍至2.0倍的加速效果。
原文摘要 · Abstract (English)
Speculative decoding is a standard method for accelerating the inference speed of large language models. However, scaling it for production environments poses several engineering challenges, including efficiently implementing different operations (e.g., tree attention and multi-round speculative decoding) on GPU. In this paper, we detail the training and inference optimization techniques that we have implemented to enable EAGLE-based speculative decoding at a production scale for Llama models. With these changes, we achieve a new state-of-the-art inference latency for Llama models. For example, Llama4 Maverick decodes at a speed of about 4 ms per token (with a batch size of one) on 8 NVIDIA H100 GPUs, which is 10% faster than the previously best known method. Furthermore, for EAGLE-based speculative decoding, our optimizations enable us to achieve a speed-up for large batch sizes between 1.4x and 2.0x at production scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。