用分位置专精的生成器提升大模型推测解码效率
POSS: Position Specialist Generates Better Draft for Speculative Decoding
- 设计多个专注特定位置的生成模块，减少误差累积
- 在后期位置上接受率显著提升，平均加速比更高
- 适合需要高速推理的大模型应用开发人员
推测解码通过小模型预测多个词元，并由大模型并行验证来加速大语言模型推理。现有方法利用目标模型隐藏状态提升小模型预测精度，但因小模型生成特征误差积累，导致后期位置预测质量下降。本文提出位置专精模块（PosS），由多个针对特定位置的生成层组成，专门处理不同层级的特征偏差。在 Llama-3-8B-Instruct 和 Llama-2-13B-chat 上跨六个数据集的实验表明，PosS 显著提升每轮推测的平均接受长度与加速比。代码已开源。
原文摘要 · Abstract (English)
Speculative decoding accelerates Large Language Model (LLM) inference by using a small draft model to predict multiple tokens, and a large target model to verify these tokens in parallel. Recent studies leverage the hidden state of the target model to enhance draft model prediction accuracy. However, existing methods suffer from the degrading quality of draft token predictions at later positions, due to error accumulation in draft model generated features. In this paper, we propose Position Specialists (PosS), which consist of multiple position-specialized draft layers to generate tokens at assigned position(s). Position specialists greatly improve token acceptance rate at later positions per drafting round, as each specialist only needs to focus on handling a certain level of draft model feature deviation. Experiment results on Llama-3-8B-Instruct and Llama-2-13B-chat across six datasets demonstrate that PosS effectively improves over baselines on average acceptance length and speed-up ratio. Our codebase is available at https://github.com/shrango/PosS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。