让大模型一次预测多个词，提升效率与性能。
Distribution-Aware Companding Quantization of Large Language Models
- 用多头并行预测后续n个词，共享主干网络
- 13B模型在HumanEval上解题率高12%，MBPP高17%
- 推理速度最高提升3倍，适合代码生成任务
大型语言模型如GPT和Llama通常采用单词预测损失进行训练。本文提出，在每个位置同时预测接下来的n个词，可显著提升样本效率。具体地，在训练语料的每个位置，使用n个独立输出头预测后续n个词，均基于共享的模型主干。将多词预测作为辅助任务，不增加训练时间，即可在代码和自然语言任务中提升下游性能。该方法对更大模型规模更有效，多轮训练下仍具优势。在生成类基准测试中表现尤为突出，我们的13B参数模型在HumanEval上解决率比基线高12%，在MBPP上高17%。小规模算法任务实验表明，多词预测有助于诱导头和算法推理能力的发展。额外优势是，经4词预测训练的模型在推理时最多快3倍，即使使用大批次。
原文摘要 · Abstract (English)
Large language models such as GPT and Llama are trained with a next-token prediction loss. In this work, we suggest that training language models to predict multiple future tokens at once results in higher sample efficiency. More specifically, at each position in the training corpus, we ask the model to predict the following n tokens using n independent output heads, operating on top of a shared model trunk. Considering multi-token prediction as an auxiliary training task, we measure improved downstream capabilities with no overhead in training time for both code and natural language models. The method is increasingly useful for larger model sizes and keeps its appeal when training for multiple epochs. Gains are especially pronounced on generative benchmarks like coding, where our models consistently outperform strong baselines by several percentage points. Our 13B parameter models solves 12 % more problems on HumanEval and 17 % more on MBPP than comparable next-token models. Experiments on small algorithmic tasks demonstrate that multi-token prediction is favorable for the development of induction heads and algorithmic reasoning capabilities. As an additional benefit, models trained with 4-token prediction are up to 3X times faster at inference, even with large batch sizes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。