通过优化多词预测提升大模型推理速度,实现无损加速。
FastMTP: Accelerating LLM Inference with Enhanced Multi-Token Prediction
- 用自蒸馏数据微调共享位置权重的MTP头,提升连续词预测质量。
- 在7个基准上平均提速2.03倍,比原MTP快82%,输出质量不变。
- 轻量训练+兼容现有框架,适合急需提速的部署场景。
随着大语言模型能力增强,自回归生成的串行特性带来了根本性的吞吐瓶颈。尽管多词预测(MTP)在训练效率和性能上表现优异,其对推理加速的潜力尚未充分挖掘。本文提出FastMTP,一种简单有效的改进方法:通过将MTP训练与推理模式对齐,提升多步草稿质量,显著增强推测解码性能。该方法在自蒸馏数据上微调单一共享位置权重的MTP头,使其捕捉未来连续词间的依赖关系,并在多层递归草稿中保持高接受率。通过在MTP头中引入语言感知的动态词汇压缩,进一步降低草稿过程的计算开销。在七个不同基准上的实验表明,FastMTP相比标准单词预测平均提速2.03倍,输出质量无损,优于原始MTP 82%。FastMTP仅需轻量级训练,可无缝集成至现有推理框架,提供一种实用且快速部署的大模型推理加速方案。
原文摘要 · Abstract (English)
As large language models (LLMs) become increasingly powerful, the sequential nature of autoregressive generation creates a fundamental throughput bottleneck that limits the practical deployment. While Multi-Token Prediction (MTP) has demonstrated remarkable benefits for model training efficiency and performance, its inherent potential for inference acceleration remains largely unexplored. This paper introduces FastMTP, a simple yet effective method that improves multi-step draft quality by aligning MTP training with its inference pattern, significantly enhancing speculative decoding performance. Our approach fine-tunes a single MTP head with position-shared weights on self-distilled data, enabling it to capture dependencies among consecutive future tokens and maintain high acceptance rates across multiple recursive draft steps. By integrating language-aware dynamic vocabulary compression into the MTP head, we further reduce computational overhead in the drafting process. Experimental results across seven diverse benchmarks demonstrate that FastMTP achieves an average of 2.03x speedup compared to standard next token prediction with lossless output quality, outperforming vanilla MTP by 82%. FastMTP requires only lightweight training and seamlessly integrates with existing inference frameworks, offering a practical and rapidly deployable solution for accelerating LLM inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。