arXiv:2510.05421cs.LG2025-10被引 2

让模型自动生成并自我优化,实现高效无损推理加速。

Draft, Verify, and Improve: Toward Training-Aware Speculative Decoding

  • 用验证结果反向训练生成器,实现在线持续学习。
  • 在Spec-Bench上提速2.16倍,训练数据量远低于同类方法。
  • 无需额外组件,适合追求高效部署的工业级应用。

自回归解码是大语言模型的主要延迟瓶颈。推测解码(SD)通过让起草器提出多标记块,由验证器接受或拒绝来加速。然而,许多现有系统依赖昂贵的离线训练或额外组件,增加数据与计算成本,且在分布漂移下易产生脆弱的起草器。本文提出训练感知的自推测框架DVI,将大模型分为起草器与验证器,在生成过程中将验证器的接受/拒绝决策转化为监督信号,用于更新起草器头部。采用简单的KL→RL调度:先通过在线蒸馏校准,再引入带奖励掩码的交叉熵和在线策略梯度项,实现无损、单模型部署。在Spec-Bench上,DVI达到2.16倍的墙时加速,性能媲美当前最优方法EAGLE-2,但训练数据量少几个数量级;消融实验表明,DVI优于仅使用KL的在线蒸馏。结果证明,训练感知的自推测可实现顶尖速度提升,且训练开销极小。

原文摘要 · Abstract (English)

Autoregressive (AR) decoding is a major latency bottleneck for large language models. Speculative decoding (SD) accelerates AR by letting a drafter propose multi-token blocks that a verifier accepts or rejects. However, many SD systems require heavy offline training or extra components. These choices raise data/compute cost and can yield brittle drafters under distribution drift. We introduce \emph{Draft, Verify, \& Improve (DVI)}, a training-aware self-speculative framework that combines inference with continual online learning. We partition an LLM into a drafter and a verifier, and during generation, verifier accept/reject decisions are converted into supervision signals and used to update the drafter head. A simple \emph{KL$\rightarrow$RL} schedule bootstraps calibration via online distillation and then adds reward-masked cross-entropy with a on-policy policy-gradient term, preserving lossless, single model deployment. On Spec-Bench, DVI achieves a $2.16\times$ wall-time speedup, on par with SoTA approaches like EAGLE-2, while orders of magnitude less data for training, and ablations show that DVI outperforms KL-only online distillation. DVI demonstrates that \emph{training-aware} self-speculation can deliver state-of-the-art, lossless speedups with minimal training overhead.

推理加速自推测在线学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。