通过自蒸馏提升大模型多标记并行预测效率
Self-Distillation for Multi-Token Prediction
- 用自蒸馏方法提升多标记预测头接受率
- 接受率提升7.5%，推理速度最高快220.4%
- 适合追求高效推理的大模型应用
随着大语言模型（LLM）规模扩大，推理效率成为关键瓶颈。多标记预测（MTP）可通过并行预测多个未来标记加速推理，但现有方法仍面临两个挑战：MTP头接受率有限，以及难以联合训练多个MTP头。为此，我们提出MTP-D，一种简单而有效的自蒸馏方法，额外训练成本极低，将MTP头接受率提升7.5%，同时最大程度保留主头性能。我们还引入循环扩展策略，实现高效经济的MTP头扩展，使1头MTP推理速度提升220.4%。在七个基准上的系统性实验验证了蒸馏策略的有效性及MTP的可扩展性。结果表明，MTP-D与循环扩展策略显著提升MTP头性能和推理效率，推动了MTP在大模型中的实际应用。
原文摘要 · Abstract (English)
As Large Language Models (LLMs) scale up, inference efficiency becomes a critical bottleneck. Multi-Token Prediction (MTP) could accelerate LLM inference by predicting multiple future tokens in parallel. However, existing MTP approaches still face two challenges: limited acceptance rates of MTP heads, and difficulties in jointly training multiple MTP heads. Therefore, we propose MTP-D, a simple yet effective self-distillation method with minimal additional training cost, which boosts MTP head acceptance rates (+7.5\%) while maximumly preserving main-head performance. We also introduce a looped extension strategy for MTP-D, enabling effective and economical MTP head extension and further significant inference speedup to 1-head MTP (+220.4\%). Moreover, we systematically explore and validate key insights on the distillation strategies and the potential scalability of MTP through extensive experiments on seven benchmarks. These results demonstrate that our MTP-D and looped extension strategy effectively enhance MTP-head performance and inference efficiency, facilitating the practical usage of MTP in LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。