动态调整生成长度，让模型自己判断何时停止草稿生成。
Draft Model Knows When to Stop: Self-Verification Speculative Decoding for Long-Form Generation
- 根据草稿模型预测熵动态决定生成长度，无需训练。
- 在8K上下文下比固定长度快17%，长文本推理快22%。
- 适合需要高效长文本生成的复杂推理场景。
传统推测解码方法采用预设长度策略生成草稿，假设目标模型能平滑接受所有草稿词元。然而现实中草稿长度需求波动大，固定长度策略难以满足。尤其在复杂推理与长文本生成任务中，这一问题更为显著，尤其是在推理专用模型的测试时缩放场景下。通过理论与实证分析，我们发现草稿模型预测熵可近似反映草稿与目标模型之间的差异：熵越高，草稿词元被接受率越低。基于此，我们提出SVIP（自验证长度策略），一种无需训练的动态长度策略，通过参考草稿熵自适应决定草稿序列长度。在主流推测解码基准及高推理强度基准上的实验表明，SVIP性能更优，在MT-Bench 8K上下文下相比固定长度最高提升17%速度，对QwQ模型在长文本推理中提速达22%。
原文摘要 · Abstract (English)
Conventional speculative decoding (SD) methods utilize a predefined length policy for proposing drafts, which implies the premise that the target model smoothly accepts the proposed draft tokens. However, reality deviates from this assumption: the oracle draft length varies significantly, and the fixed-length policy hardly satisfies such a requirement. Moreover, such discrepancy is further exacerbated in scenarios involving complex reasoning and long-form generation, particularly under test-time scaling for reasoning-specialized models. Through both theoretical and empirical estimation, we establish that the discrepancy between the draft and target models can be approximated by the draft model's prediction entropy: a high entropy indicates a low acceptance rate of draft tokens, and vice versa. Based on this insight, we propose SVIP: Self-Verification Length Policy for Long-Context Speculative Decoding, which is a training-free dynamic length policy for speculative decoding systems that adaptively determines the lengths of draft sequences by referring to the draft entropy. Experimental results on mainstream SD benchmarks as well as reasoning-heavy benchmarks demonstrate the superior performance of SVIP, achieving up to 17% speedup on MT-Bench at 8K context compared with fixed draft lengths, and 22% speedup for QwQ in long-form reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。