arXiv:2505.18629cs.LG2025-05被引 6

用大模型自我反思机制,让快速生成的文本更准确。

Think Before You Accept: Semantic Reflective Verification for Faster Speculative Decoding

  • 利用大模型自省能力,单次前向传播完成语义验证
  • 在多个基准上提升草案词接受长度,速度最高快15%
  • 无需训练,兼容现有方法,适合追求高效推理的研究者

大型语言模型因自回归解码导致推理延迟高。推测解码通过轻量模型生成多个草案词并行验证以加速。然而,现有验证方法过度依赖分布一致性,忽视语义正确性,限制了提速潜力。部分方法虽引入额外模型进行宽松验证,但难以泛化至多样或开放域场景。本文提出无训练、语义感知的反射验证(Reflective Verification),充分利用大模型的内在自省能力,在验证阶段并行评估草案词的语义正确性。通过提示词探测,单次前向传播获取草案词的原始与反射分布,融合后实现语义层面的验证,兼顾一致性与正确性。跨多领域基准和模型规模的实验表明,该方法显著提升草案词接受长度,且不降低模型性能。进一步发现,该方法与现有统计验证方法正交,二者结合可带来5~15%的额外解码速度提升。

原文摘要 · Abstract (English)

Large language models (LLMs) suffer from high inference latency due to the auto-regressive decoding process. Speculative decoding accelerates inference by generating multiple draft tokens using a lightweight model and verifying them in parallel. However, existing verification methods rely heavily on distributional consistency while overlooking semantic correctness, thereby limiting the potential speedup of speculative decoding. While some methods employ additional models for relaxed verification of draft tokens, they often fail to generalize effectively to more diverse or open-domain settings. In this work, we propose Reflective Verification, a training-free and semantics-aware approach that achieves a better trade-off between correctness and efficiency. Specifically, we leverage the inherent reflective capacity of LLMs to semantically assess the correctness of draft tokens in parallel during verification. Using prompt-based probing, we obtain both the original and reflective distributions of draft tokens in a single forward pass. The fusion of these distributions enables semantic-level verification of draft tokens that incorporates both consistency and correctness. Experiments across multiple domain benchmarks and model scales demonstrate that our method significantly increases the acceptance length of draft tokens without compromising model performance. Furthermore, we find that the proposed Reflective Verification is orthogonal to existing statistical verification methods, and their combination yields additional 5$\sim$15\% improvements in decoding speed.

推理加速语义验证推测解码

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。