让小模型更懂大模型：用新方法训练的猜词模型，推理快6.94倍
PARD-2: Target-Aligned Parallel Draft Model for Dual-Mode Speculative Decoding
- 改用接受长度优化目标，让小模型猜词更符合大模型验证逻辑
- 支持两种模式，单模型实现更快且无损失加速，最高提速6.94倍
- 适合追求高效推理的模型部署者，尤其在资源受限场景
推测解码通过轻量级草稿模型并行生成候选词，由目标模型验证以加速大语言模型推理。现有草稿模型训练目标与推理阶段最大化连续词接受率不一致。为此，本文重构草稿模型优化目标，从预测准确率转向整体接受长度。基于PARD提出PARD-2，一种双模式推测解码框架，采用置信度自适应词元（CAT）优化，动态调整每词权重以更好匹配验证过程。显著地，PARD-2使单一草稿模型同时支持目标相关与无关模式。在多种模型和任务上实验表明，其在Llama3.1-8B上实现高达6.94×的无损加速，超越EAGLE-3达1.9×，优于PARD 1.3×。代码已开源。
原文摘要 · Abstract (English)
Speculative decoding accelerates Large Language Models (LLMs) inference by using a lightweight draft model to propose candidate tokens that are verified in parallel by the target model. However, existing draft model training objectives are not directly aligned with the inference-time goal of maximizing consecutive token acceptance. To address this issue, we reformulate the draft model optimization objective, shifting the focus from token prediction accuracy to the overall acceptance length. In this paper, we build upon PARD to propose PARD-2, a dual-mode speculative decoding framework with Confidence-Adaptive Token (CAT) optimization. This approach adaptively reweights each token to better align with the verification process. Notably, PARD-2 enables a single draft model to support both target-dependent and target-independent modes. Experiments across diverse models and tasks demonstrate that PARD-2 achieves up to 6.94$\times$ lossless acceleration, surpassing EAGLE-3 by 1.9$\times$ and PARD by 1.3$\times$ on Llama3.1-8B. Our code is available at https://github.com/AMD-AGI/PARD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。