探索大模型并行生成多个词元的潜力，提升推理效率。
On multi-token prediction for efficient LLM inference
- 利用中间词元概率数值积分实现多词元预测。
- 模型规模越大，多词元预测效果越佳，但受数据影响明显。
- 冻结模型难适配多词元头，需新训练策略支持。
我们系统研究了预训练为单词元预测（NTP）的大语言模型在多词元预测（MTP）方面的能力。结果显示，通过数值积分中间词元概率，此类模型天然具备MTP能力，但性能依赖于数据，且随模型规模提升而改善。此外，我们将MTP头集成到冻结的LLM中时发现，其隐藏层高度专业化于NTP，适应过程困难。最后，联合训练MTP头虽可提升性能，但仍无法完全突破该瓶颈，提示需进一步研究。本研究深化了对预训练模型中MTP应用的理解，为通过并行词元预测加速推理提供策略参考。
原文摘要 · Abstract (English)
We systematically investigate multi-token prediction (MTP) capabilities within LLMs pre-trained for next-token prediction (NTP). We first show that such models inherently possess MTP capabilities via numerical marginalization over intermediate token probabilities, though performance is data-dependent and improves with model scale. Furthermore, we explore the challenges of integrating MTP heads into frozen LLMs and find that their hidden layers are strongly specialized for NTP, making adaptation non-trivial. Finally, we show that while joint training of MTP heads with the backbone improves performance, it cannot fully overcome this barrier, prompting further research in this direction. Our findings provide a deeper understanding of MTP applied to pretrained LLMs, informing strategies for accelerating inference through parallel token prediction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。