不训练模型也能加速大模型生成，但需权衡速度与准确性。
A Practical Investigation of Training-free Relaxed Speculative Decoding
- 用快速辅助模型预写候选词，再由主模型并行验证。
- 放松严格校验可提速，但可能损失生成质量。
- 适合追求效率的开发者，不适用于轻量级预写器。
推测解码通过使用更快速的辅助模型预先生成候选标记，并由主语言模型并行验证，从而加速自回归大模型的采样过程。标准推测解码是无损的：其拒绝与重采样步骤能精确保持主模型的采样分布。近期研究指出，放宽这一严格保证可带来进一步提速、可控的能力-速度权衡，甚至能力提升。本文对无需训练的松弛型推测解码技术进行了实际考察，将现有方法统一于同一框架下，在现代场景中进行基准测试，并为实践者提炼出关键经验。重要发现包括：松弛策略需要显著的能力评估，且多数方法依赖性能良好的语言模型作为预写器，因此不适用于轻量级的多标记预测专用预写器。
原文摘要 · Abstract (English)
Speculative decoding accelerates sampling from an autoregressive LLM by using a faster auxiliary model to draft tokens which are then verified in parallel by the LLM. Standard speculative decoding is lossless: its rejection and resampling steps exactly preserve the LLM's sampling distribution. Recent work argues that relaxing this strict guarantee can yield further speed-ups, controlled capability-speed trade-offs, or even capability gains. We practically investigate training-free relaxed speculative decoding techniques, unify existing approaches within a shared framework, benchmark them on contemporary settings, and distil takeaways and empirical findings for practitioners. Important takeaways include: relaxation can require considerable capability evaluation unlike lossless speculative decoding, and many relaxed approaches rely on a drafter that is a good language model, making them unsuited for lightweight dedicated multi-token-prediction drafters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。