提出Draft-OPD方法,让轻量模型更高效地生成可接受的候选文本。
Draft-OPD: On-Policy Distillation for Speculative Draft Models

- 用目标模型监督轻量模型在自身生成状态下的表现,实现在线策略蒸馏。
- 实验显示在多种任务上实现超过5倍的无损加速,优于EAGLE-3和DFlash。
- 适合追求推理加速且关注生成质量的LLM应用开发者。
推测解码通过配对一个目标模型与轻量级草稿模型来加速大语言模型推理,草稿模型提出的词元在并行中被验证。常见构建草稿模型的方法(如EAGLE3或DFlash)是基于目标模型生成轨迹进行监督微调(SFT)。然而我们发现SFT很快达到瓶颈:测试数据上的草稿模型接受长度不再提升。原因是离线训练与在线推理不匹配——在SFT中,草稿模型学习固定的目标生成轨迹,而推测解码时评估的是其自身策略下提出的块。为此我们提出在线策略蒸馏(OPD),由目标模型在草稿诱导的状态上监督草稿模型。但草稿模型难以独立完成完整序列生成,而目标辅助生成使收集序列符合目标分布,从而消除在线策略信号。因此我们提出Draft-OPD,利用目标辅助回滚实现稳定续写,并从验证暴露的错误位置重放草稿过程。这使得草稿模型能从目标反馈中学习接受与拒绝的提案,重点训练影响推测接受率的草稿误差。实验表明,Draft-OPD在多种任务上实现超过5×的无损加速,相较于EAGLE-3和DFlash分别提升23%和13%。
原文摘要 · Abstract (English)
Speculative decoding accelerates large language model inference by pairing a target model with a lightweight draft model whose proposed tokens are verified in parallel. A common way to build draft models, like EAGLE3 or DFlash is supervised fine-tuning (SFT) on target-generated trajectories. However, we observe that SFT quickly plateaus: the draft model's acceptance length on test data stops improving. The reason is an offline-to-inference mismatch: In SFT, the drafter learns from fixed target-generated trajectories, whereas during speculative decoding it is evaluated on blocks proposed under its own policy. This motivates on-policy distillation (OPD), where the target model supervises the drafter on draft-induced states. Yet OPD remains difficult for draft models, as they cannot reliably roll out complete sequences independently, whereas target-assisted generation makes the collected sequences follow the target distribution and thus eliminates the on-policy signal. We therefore propose Draft-OPD, which uses target-assisted rollout for stable continuations and replays drafting from the verification-exposed error positions. This allows the drafter to learn from target feedback on both accepted and rejected proposals, focusing training on the draft-induced errors that limit speculative acceptance. Experiments show that Draft-OPD achieves over $5\times$ lossless acceleration for thinking models across diverse tasks, improving over EAGLE-3 and DFlash by 23\% and 13\%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。