用重要性重加权提升大模型推理速度，不丢精度。
Faster LLM Inference via Sequential Monte Carlo
- 用粒子采样与权重重估替代逐字拒绝，实现并行验证。
- 推理速度比传统推测解码快2.36倍，比自回归快5.2倍。
- 适合追求高吞吐、低延迟的部署场景，尤其在内存受限时
推测解码（SD）通过廉价的草稿模型生成候选词元，并用昂贵的目标模型通过拒绝采样进行验证，从而加速语言模型推理。由于拒绝会立即截断首个错误的草稿块，当草稿与目标模型产生分歧时，吞吐量会下降。本文提出一种基于序贯蒙特卡洛的推测解码（SMC-SD），将词元级拒绝替换为草稿粒子群体的重要性加权重采样。该方法是一种有理论保障的近似推理方案，在牺牲部分精确度的同时显著提升速度，且每步近似误差具有可证明的上界。由于大模型推理受内存带宽限制，草稿粒子生成与并行评分所消耗的计算几乎免费——SMC-SD利用闲置算力将验证转为固定大小的向量化操作，无需回滚。实验表明，SMC-SD相比推测解码提速2.36倍，相比自回归解码提速5.2倍，同时在推理、指令遵循和编码基准上保持与目标模型相差不超过3%的准确率。
原文摘要 · Abstract (English)
Speculative decoding (SD) accelerates language model inference by drafting tokens from a cheap proposal model and verifying them against an expensive target model via rejection sampling. Because rejection truncates the draft block at the first error, throughput degrades when draft and target diverge. Rather than rejecting draft tokens outright, we propose to reweight them. To this end, we introduce sequential Monte Carlo speculative decoding (SMC-SD), which replaces token-level rejection with importance-weighted resampling over a population of draft particles. SMC-SD is a principled approximate inference scheme that trades exactness for additional speed, while preserving theoretical bounds on its per-step approximation error. Because LLM inference is memory bandwidth-bound, the arithmetic needed to draft particles and to score them in parallel comes nearly for free -- SMC-SD uses idle compute to turn verification into a vectorized, fixed-size operation with no rollback. Empirically, SMC-SD achieves 2.36x speed-up over speculative decoding and a 5.2x speed-up over autoregressive decoding, while remaining within 3% of the target model's accuracy on reasoning, instruction-following, and coding benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。