EAGLE-3通过直接预测令牌提升大模型推理速度，支持数据量扩大时性能持续增长。
EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test
- 放弃特征预测，改用多层特征融合的训练时测试策略
- 在5个任务上实现最高6.5倍加速，较EAGLE-2提升1.4倍
- 适合追求高吞吐、可扩展训练的大模型部署场景
现代大语言模型的自回归特性导致推理成本高、速度慢，推测采样已被证明是有效解决方案。EAGLE等方法在特征层面进行自回归，复用目标模型顶层特征，优于原始推测采样。当前大模型社区普遍通过扩大训练数据来提升智能水平，而不增加推理开销。然而我们发现，数据规模扩大对EAGLE的性能提升有限。究其原因在于其特征预测机制存在约束。本文提出EAGLE-3，摒弃特征预测，转而直接预测令牌，并引入训练时测试技术实现多层特征融合，替代对顶层特征的依赖。实验涵盖对话与推理模型，在五个任务上验证：EAGLE-3实现最高6.5倍加速，相较EAGLE-2提升约1.4倍；在SGLang框架中，批量大小为64时，吞吐量提升1.38倍。代码已开源。
原文摘要 · Abstract (English)
The sequential nature of modern LLMs makes them expensive and slow, and speculative sampling has proven to be an effective solution to this problem. Methods like EAGLE perform autoregression at the feature level, reusing top-layer features from the target model to achieve better results than vanilla speculative sampling. A growing trend in the LLM community is scaling up training data to improve model intelligence without increasing inference costs. However, we observe that scaling up data provides limited improvements for EAGLE. We identify that this limitation arises from EAGLE's feature prediction constraints. In this paper, we introduce EAGLE-3, which abandons feature prediction in favor of direct token prediction and replaces reliance on top-layer features with multi-layer feature fusion via a technique named training-time test. These improvements significantly enhance performance and enable the draft model to fully benefit from scaling up training data. Our experiments include both chat models and reasoning models, evaluated on five tasks. The results show that EAGLE-3 achieves a speedup ratio up to 6.5x, with about 1.4x improvement over EAGLE-2. In the SGLang framework, EAGLE-3 achieves a 1.38x throughput improvement at a batch size of 64. The code is available at https://github.com/SafeAILab/EAGLE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。