arXiv:2511.09844cs.LGcs.PF2025-11AAAI被引 2

用轻量级向量动态对齐预训练草稿模型,提升生成效率

Steering Pretrained Drafters during Speculative Decoding

  • 基于验证器隐状态生成转向向量,实时调整草稿模型输出
  • 标准采样下可多接受35%的令牌,贪婪采样下提升22%
  • 无需重新训练,可直接部署到现有模型架构中

推测解码通过将生成分为快速草稿和并行验证两阶段来加速语言模型推理。其主要瓶颈在于草稿与验证之间的不一致,限制了令牌接受率,降低整体效率。虽然从头训练的小型草稿头速度快,但在验证主导延迟或输入分布外时表现不佳。相比之下,预训练草稿头虽慢,但凭借更强的独立生成能力,当草稿延迟相对于验证或通信开销可忽略时更具竞争力。本文提出一种轻量级动态对齐机制:从验证器隐状态计算转向向量,并注入预训练草稿头。相比现有离线对齐方法(如知识蒸馏),该方法在标准采样下令牌接受率提升最高达35%,贪婪采样下提升22%,且计算开销可忽略。更重要的是,该方法可无缝适配现有架构与预训练模型,实现快速部署。

原文摘要 · Abstract (English)

Speculative decoding accelerates language model inference by separating generation into fast drafting and parallel verification. Its main limitation is drafter-verifier misalignment, which limits token acceptance and reduces overall effectiveness. While small drafting heads trained from scratch compensate with speed, they struggle when verification dominates latency or when inputs are out of distribution. In contrast, pretrained drafters, though slower, achieve higher acceptance rates thanks to stronger standalone generation capabilities, making them competitive when drafting latency is negligible relative to verification or communication overhead. In this work, we aim to improve the acceptance rates of pretrained drafters by introducing a lightweight dynamic alignment mechanism: a steering vector computed from the verifier's hidden states and injected into the pretrained drafter. Compared to existing offline alignment methods such as distillation, our approach boosts the number of accepted tokens by up to 35\% under standard sampling and 22\% under greedy sampling, all while incurring negligible computational overhead. Importantly, our approach can be retrofitted to existing architectures and pretrained models, enabling rapid adoption.

推理加速推测解码模型对齐轻量级

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。