解决大模型推测解码中的分词错位问题，提升生成速度与准确性。
GRIFFIN: Effective Token Alignment for Faster Speculative Decoding
- 通过训练时屏蔽错位分词，优化生成一致性
- 在多个主流模型上实现平均接受长度提升8%以上
- 适合追求推理加速的LLM应用开发者
推测解码通过并行生成多个草稿分词来加速大语言模型（LLMs）的推理，但现有方法常因训练与解码阶段的分词错位而受限。为此，我们提出GRIFFIN框架，包含可对齐的训练策略和可对齐的草稿模型，以缓解错位问题。训练策略采用损失掩码机制，在训练中排除高度错位的分词，避免其对草稿模型优化产生负面影响。可对齐的草稿模型引入输入分词以修正生成特征的不一致。在LLaMA、Vicuna、Qwen和Mixtral模型上的实验表明，GRIFFIN实现了平均接受长度超过8%的提升和超过7%的速度提升，优于当前最先进的推测解码方法。代码与GRIFFIN草稿模型已公开于https://github.com/hsj576/GRIFFIN。
原文摘要 · Abstract (English)
Speculative decoding accelerates inference in large language models (LLMs) by generating multiple draft tokens simultaneously. However, existing methods often struggle with token misalignment between the training and decoding phases, limiting their performance. To address this, we propose GRIFFIN, a novel framework that incorporates a token-alignable training strategy and a token-alignable draft model to mitigate misalignment. The training strategy employs a loss masking mechanism to exclude highly misaligned tokens during training, preventing them from negatively impacting the draft model's optimization. The token-alignable draft model introduces input tokens to correct inconsistencies in generated features. Experiments on LLaMA, Vicuna, Qwen and Mixtral models demonstrate that GRIFFIN achieves an average acceptance length improvement of over 8% and a speedup ratio exceeding 7%, outperforming current speculative decoding state-of-the-art methods. Our code and GRIFFIN's draft models are released publicly in https://github.com/hsj576/GRIFFIN.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。