通过变分推理优化生成路径，提升大模型推理速度
Variational Speculative Decoding: Rethinking Draft Training from Token Likelihood to Sequence Acceptance
- 将草稿训练建模为潜变量推断，最大化目标模型接受概率
- 实验显示相比EAGLE-3提速9.6%，ViSpec提速7.9%
- 适合追求高效推理的LLM/MML应用开发者
推测解码可加速（多）大语言模型的推理，但训练与解码之间存在不一致：现有方法仅优化单一贪婪轨迹，而实际解码需验证和排序多个采样草稿路径。我们提出变分推测解码（VSD），将草稿训练形式化为对潜在提案（草稿路径）的变分推断。VSD最大化目标模型接受的边缘概率，得到一个促进高质量潜变量提案并最小化与目标分布差异的证据下界（ELBO）。为提升质量并降低方差，引入路径级效用，并通过期望-最大化（EM）过程优化。E步从由预言机筛选的后验中进行蒙特卡洛采样，M步则利用自适应拒绝加权（ARW）和置信度感知正则化（CAR）最大化加权似然。理论分析表明，VSD可增加预期接受长度和加速比。在多种大模型和多模态大模型上的大量实验表明，VSD相比EAGLE-3最高提速9.6%，相比ViSpec提速7.9%，显著提升解码效率。
原文摘要 · Abstract (English)
Speculative decoding accelerates inference for (M)LLMs, yet a training-decoding discrepancy persists: while existing methods optimize single greedy trajectories, decoding involves verifying and ranking multiple sampled draft paths. We propose Variational Speculative Decoding (VSD), formulating draft training as variational inference over latent proposals (draft paths). VSD maximizes the marginal probability of target-model acceptance, yielding an ELBO that promotes high-quality latent proposals while minimizing divergence from the target distribution. To enhance quality and reduce variance, we incorporate a path-level utility and optimize via an Expectation-Maximization procedure. The E-step draws Monte Carlo samples from an oracle-filtered posterior, while the M-step maximizes weighted likelihood using Adaptive Rejection Weighting (ARW) and Confidence-Aware Regularization (CAR). Theoretical analysis confirms that VSD increases expected acceptance length and speedup. Extensive experiments across LLMs and MLLMs show that VSD achieves up to a 9.6% speedup over EAGLE-3 and 7.9% over ViSpec, significantly improving decoding efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。