分析了推测解码中损失验证的失效机制,揭示其对生成质量的影响。
Revisiting Lossy Verification in Speculative Decoding: Mechanisms, Trade-offs, and Failure Modes

- 将损失验证分为截断与协作两类,发现其本质差异
- 截断类方法因分布偏移导致生成质量严重下降
- 协作类方法需控制草稿概率超调以保证输出质量
推测解码(SD)通过轻量级草稿模型提出候选词,并由大型目标模型并行验证以加速大语言模型推理。近期方法引入损失验证机制,通过放宽分布匹配要求进一步提升效率,但此类放松会悄然改变解码分布,可能导致生成质量不稳定甚至显著下降。本文对损失验证引发的分布进行系统分析,发现看似不同的方法实则可归为两类:基于截断的验证与协作式验证。我们构建了精选基准的诊断评估框架。针对截断类方法,发现其性能可能远低于真实截断采样基线,源于分布畸变;针对协作类方法,揭示关键原则:必须控制草稿概率相对于目标概率的超调,否则会产生低质量输出。代码已公开于 https://github.com/ZhouYuxuanYX/Fast-HSD。
原文摘要 · Abstract (English)
Speculative Decoding (SD) accelerates large language model inference by allowing a lightweight draft model to propose tokens that are subsequently verified in parallel by a larger target model. Recent approaches introduce lossy verification schemes to further improve efficiency by relaxing strict distributional matching. Yet such relaxation silently rewrites the decoding distribution, and the resulting acceleration can come at the cost of unstable, sometimes severely degraded generation quality. In this work, we present a principled analysis of the distributions induced by lossy verification methods. We show that many seemingly distinct approaches differ only superficially and can be classified into two categories: truncation-based verification and collaborative verification. We further construct a diagnostic evaluation framework across curated benchmarks. For truncation-based methods, we identify a fundamental pitfall: performance can degrade significantly compared to the true truncation sampling baseline due to distributional distortion. For collaborative verification, we uncover a key principles: controlling the overshoot of draft probabilities relative to target probabilities is essential to prevent low-quality outputs. Our code is available at https://github.com/ZhouYuxuanYX/Fast-HSD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。