让并行草案更连贯,提升大模型推理速度
LiLiCorr: Lightweight Likelihood Correlation of Parallel Drafts for Speculative Decoding
- 用轻量模型关联草案各位置的候选词分布,捕捉整体结构
- 相比原版提升9%-19%的接受长度,仅增加2.8%延迟
- 适合追求高速推理的部署场景,尤其长文本生成
推测解码通过并行生成未来标记候选来加速语言模型推理。以DFlash为代表的扩散式块头可一次性预测整个标记块,但其训练基于单位置边缘分布而非联合块分布,导致生成的标记虽单个合理却整体不连贯。本文提出LiLiCorr,一种基于似然的轻量级相关模型,用于关联草案已产生的各位置边缘分布。它保留每个位置的top-k候选,并联合处理,为每个候选生成in和out向量。相邻候选匹配标准为前一候选的out向量与后一候选的in向量余弦相似度高。该匹配机制捕获了标记块的联合结构,而无需显式构建完整联合分布。一次轻量网络计算即可生成所有向量,成对评分通过批矩阵运算并行完成,仅需廉价的贪心遍历串行处理。进一步与草案器联合训练,使其学会生成能形成更长可接受序列的候选。在所有基准测试中,相比原版DFlash,LiLiCorr将接受长度提升9%至19%,其评分头仅占每块延迟的约2.8%。相较于DFlash及另外两种同期方法,LiLiCorr在72组设置中的70组实现了最高吞吐量:九个基准、两种目标模型大小下采用贪婪与temperature-one解码,以及覆盖六种并发数、两种输入长度、三种熵级别的吞吐量扫描,所有系统均在同一服务栈上等效优化。将LiLiCorr扩展至比训练时长一个数量级的输入,仍保持领先优势。
原文摘要 · Abstract (English)
Speculative decoding accelerates language-model inference by drafting future tokens that the target model verifies in parallel. A diffusion-style block head such as DFlash is an attractive drafter, predicting an entire block of future tokens in one forward pass. However, it is trained on per-position marginals rather than the joint block distribution, so the tokens it emits are individually plausible yet jointly incoherent. We introduce LiLiCorr, a Lightweight Likelihood-based model that Correlates the per-position marginal distributions a drafter already produces. It keeps the top-k tokens at each position as candidates and processes them jointly, producing for each an in and an out vector. A pair of adjacent candidates matches when the earlier one's out vector has high cosine similarity with the later one's in vector. These matches capture the block's joint structure without ever materializing the full joint distribution. One lightweight network pass produces all the vectors, and the pairwise scores are then computed in parallel as batched matrix operations, leaving only a cheap greedy walk sequential. We further co-train the drafter with LiLiCorr, so it learns to propose candidates that correlate into longer accepted sequences. Over the vanilla DFlash drafter, LiLiCorr raises acceptance length on every benchmark by 9 to 19%, while its scoring head accounts for about 2.8% of the per-block latency. Against DFlash and two concurrent methods that also restore coherence at draft time, LiLiCorr delivers the highest throughput in 70 of 72 settings: nine benchmarks at two target sizes under greedy and temperature-one decoding, and a throughput sweep over six concurrencies, two input lengths and three entropy tiers, with all systems equally optimized on a common serving stack. Extending LiLiCorr to inputs an order of magnitude longer than it was trained on preserves that lead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。