用多个小模型协作加速大模型生成，自动选最优方案。
Multi-Drafter Speculative Decoding with Alignment Feedback
- 多个小模型协同出草案，根据匹配度动态分配算力。
- 在多个数据集上比单一草案模型快1.8~2.3倍，保持生成质量。
- 适合需要高速推理的场景，如实时对话系统。
推测解码（SD）通过使用小型模型预先生成未来词元，由目标大模型验证以加速推理过程，确保生成质量。然而，单个草案模型通常仅针对特定任务或领域训练，泛化能力有限。为此，我们提出 extsc{MetaSD}，一个将多个异构草案模型统一整合到推测解码框架中的新方法。该方法利用对齐反馈动态分配计算资源，并将草案选择建模为多臂赌博机问题。大量实验表明，MetaSD 在多种任务中均显著优于单一草案模型方案。
原文摘要 · Abstract (English)
Speculative decoding (SD) accelerates large language model (LLM) inference by using a smaller model to draft future tokens, which are then verified by the target LLM. This preserves generation quality by accepting only aligned tokens. However, individual drafters, often trained for specific tasks or domains, exhibit limited effectiveness across diverse applications. To address this, we introduce \textsc{MetaSD}, a unified framework that integrates multiple drafters into the SD process. MetaSD dynamically allocates computational resources to heterogeneous drafters by leveraging alignment feedback and framing drafter selection as a multi-armed bandit problem. Extensive experiments show MetaSD consistently outperforms single-drafter approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。