用自蒸馏让语言模型一次预测多个词，速度提升3倍以上。
Multi-Token Prediction via Self-Distillation
- 通过在线自蒸馏将单步预测模型转为多词并行生成。
- 在GSM8K上速度提升超3倍，准确率仅下降不足5%。
- 无需额外模型或复杂推理代码，直接部署即可用。
现有加速语言模型推理的技术（如推测解码）需要训练辅助模型并构建复杂的推理流程。本文提出一种新方法，利用简单的在线蒸馏目标，将预训练的自回归语言模型从慢速的单次词预测模式转换为快速的独立多词预测模式。最终模型与原始检查点保持完全相同的实现方式，无需添加任何辅助验证器或专用推理代码即可部署。该方法生成的模型在GSM8K数据集上的推理速度超过原模型3倍，同时准确率相对于同一检查点的单词解码性能下降小于5%。
原文摘要 · Abstract (English)
Existing techniques for accelerating language model inference, such as speculative decoding, require training auxiliary speculator models and building and deploying complex inference pipelines. We consider a new approach for converting a pretrained autoregressive language model from a slow single next token prediction model into a fast standalone multi-token prediction model using a simple online distillation objective. The final model retains the exact same implementation as the pretrained initial checkpoint and is deployable without the addition of any auxiliary verifier or other specialized inference code. Our method produces models that decode more than $3\times$ faster at $<5\%$ drop in accuracy on GSM8K relative to the single token decoding performance of the same checkpoint.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。