提出CARD框架，让大模型推理速度提升近5倍，无需额外训练。
CARD: A Cache-Assisted Parallel Speculative Decoding Framework via Query-and-Correct Paradigm for Accelerating LLM Inference
- 用查询-修正机制解耦生成与验证，支持并行加速。
- 实验显示比传统自回归推理快4.83倍，且无需模型微调。
- 适合追求高效推理的部署场景，尤其适用于资源受限环境。
推测性解码（SD）通过草稿模型并行生成多个候选词供目标模型验证，显著提升了大模型推理速度。然而，现有方法严格遵循先草稿后验证的顺序流程，限制了性能，并导致草稿中任一词被拒绝时后续所有候选词失效，造成计算浪费。为此，我们提出一种基于缓存的并行推测解码框架CARD，采用创新的查询-修正范式。该方法将草稿与验证解耦：草稿模型将候选词填充至共享缓存，目标模型则并发优化草稿路径。此设计实现接近草稿速度的推理，充分挖掘草稿模型效率，且无需对任一模型进行额外微调。实验表明，CARD显著优于现有最先进方法，在不需微调的情况下，相比原始自回归解码最高提速4.83倍。
原文摘要 · Abstract (English)
Speculative decoding (SD), where a draft model provides multiple candidate tokens for the target model to verify in parallel, has demonstrated significant potential for accelerating LLM inference. Yet, existing SD approaches adhere to a strict draft-then-verify paradigm, enforcing a sequential process that hampers performance and constrains the draft model's capacity. Moreover, rejecting a token in the candidate sequence invalidates all subsequent tokens, leading to wasted computation during drafting. To overcome these limitations, we propose a cache-assisted parallel speculative decoding framework called CARD, which employs a novel query-and-correct paradigm. Our approach decouples drafting from verification: the draft model populates a shared cache with candidate tokens, while the target model concurrently refines the draft's trajectory. This enables inference at near-draft-speed, effectively leveraging the draft model's efficiency without additional fine-tuning. Experimental results show that CARD significantly outperforms existing state-of-the-art methods, achieving up to a 4.83x acceleration over vanilla autoregressive decoding, with no fine-tuning required for either models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。