通过软硬件协同调度，实现高效多序列推测解码。
DuoDecoding: Hardware-aware Heterogeneous Speculative Decoding with Dynamic Multi-Sequence Drafting
- CPU与GPU并行部署草案与目标模型，减少等待时间。
- 动态多序列草案使生成延迟降低至原有2.61倍速。
- 适合追求低首令牌延迟的实时大模型推理场景。
大型语言模型（LLMs）在多种任务中表现卓越，但其逐标记自回归生成过程严重限制了推理速度。推测解码提供了一种先草拟后验证的框架，可在保持输出分布一致性的前提下降低生成延迟。然而，草案模型引入额外计算开销，成为性能瓶颈并增加首个标记时间（TTFT）。此前方法主要依赖启发式策略，通常难以匹配草案语言模型的质量。为此，我们提出DuoDecoding，一种新型方法：将草案模型与目标模型分别部署于CPU与GPU，实现并行解码，同时保持草案质量。该方法采用硬件感知的最优草案预算以最小化空闲时间，并引入动态多序列草案机制提升草案质量。在七个任务上的大量实验表明，DuoDecoding在生成延迟上最高实现2.61倍加速，同时将TTFT降至传统推测解码的83%。代码已开源：https://github.com/KaiLv69/DuoDecoding。
原文摘要 · Abstract (English)
Large language models (LLMs) exhibit exceptional performance across a wide range of tasks; however, their token-by-token autoregressive generation process significantly hinders inference speed. Speculative decoding presents a promising draft-then-verify framework that reduces generation latency while maintaining output distribution fidelity. Nevertheless, the draft model introduces additional computational overhead, becoming a performance bottleneck and increasing the time to first token (TTFT). Previous approaches to mitigate draft model overhead have primarily relied on heuristics and generally failed to match the quality of the draft language models. To address these challenges, we propose DuoDecoding, a novel approach that strategically deploys the draft and target models on the CPU and GPU respectively, enabling parallel decoding while preserving draft quality. Our method incorporates a hardware-aware optimal draft budget to minimize idle times and employs dynamic multi-sequence drafting to enhance draft quality. Extensive experiments across seven tasks show that DuoDecoding achieves up to 2.61x speedup in generation latency, while reducing TTFT to 83% of that in conventional speculative decoding. The Code is available at https://github.com/KaiLv69/DuoDecoding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。