根据文本可预测性动态调整生成速度，提升大模型推理效率
EntMTP: Accelerating LLM Inference with Entropy Guided Multi Token Prediction
- 基于局部熵值切换不同深度的多词预测结构，匹配上下文可预测性
- 在多个基准上实现1.15倍至1.36倍的推理加速，质量无损失
- 无需训练即可部署，适合追求高效推理的实用场景
多词预测能提升训练数据密度、改善下游文本生成质量，并成为自推测解码的标准方法。现有基础与开源模型在使用多词预测头时，始终采用静态树状注意力拓扑，导致推测深度和验证计算量在整个生成序列中保持不变，与自然语言中低熵区域可支持可靠多步推测、高熵区域需保守推测的规律不符。为此，我们提出无需训练的熵引导多词预测（EntMTP）调度器，根据运行中的局部生成熵，在一组任务特定的帕累托最优树结构间动态切换注意力拓扑。通过将推测深度与上下文可预测性对齐，EntMTP在全文本分布下最大化预期采纳词元吞吐量，且不牺牲生成质量。在Humaneval、ShareGPT、GSM8k和Litbench基准上，其相比Hydra平均提速1.15倍，相比Medusa峰值提速达1.36倍。
原文摘要 · Abstract (English)
Multi-token prediction has been shown to increase data density during training, improve downstream text-generation quality, and serves as the defacto approach for self-speculative decoding. Existing foundation and open source models that use MTP heads commit to a static tree-based attention topology throughout the entire generation sequence, meaning the speculation depth, and thus the compute required during verification, stays constant regardless of the context. This is fundamentally misaligned with the entropy patterns of natural language where low-entropy regions often support reliable multi-step drafting, while high-entropy regions require more conservative speculation. To address this, we propose Entropy-guided Multi-Token Prediction (EntMTP), a training-free scheduler that toggles between tree-based attention topologies from a set of task-specific pareto-optimal trees conditioned on a running estimate of local generation entropy. By matching speculation depth to context predictability, EntMTP maximizes expected accepted-token throughput across the full distribution of generated text without sacrificing generation quality. When evaluated across Humaneval, ShareGPT, GSM8k, and Litbench benchmarks, EntMTP consistently achieves a 1.15x speedup against Hydra and peak speedup of 1.36x against Medusa baselines respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。