用智能决策替代盲目检索,让大模型推理更快更准
When, What, and How: Rethinking Retrieval-Enhanced Speculative Decoding
- 根据上下文不确定度决定是否触发检索,避免无效推测
- 利用历史反馈筛选高质量候选,提升检索利用率
- 对生成稿严格验证,对检索稿放宽标准,兼顾速度与准确
推测解码(SD)是加速大语言模型推理的有效技术。然而其加速效果高度依赖草稿模型的性能。现有基于模型的方法如EAGLE-2虽准确但成本高,而基于检索的方法如SAM-Decoding依赖启发式切换策略,常引发不必要的检索。为此,我们提出ReSpec框架,将启发式草稿切换转化为自适应决策。ReSpec包含三项核心创新:1)基于熵的自适应触发机制,仅在上下文可预测性高时启动检索,避免低质量推测;2)基于反馈的候选选择机制,利用历史反馈组织多个高质量候选并行验证,最大化检索价值;3)源感知的宽松验证策略,对模型生成草稿施加严格检查,对检索草稿采用宽松验证,平衡准确性与效率。在Spec-Bench上的大量实验表明,ReSpec实现业界领先的加速效果,相比EAGLE-2和SAM-Decoding分别提升超过33%和25%,同时保持输出质量。
原文摘要 · Abstract (English)
Speculative decoding (SD) has emerged as an effective technique to accelerate large language model (LLM) inference without compromising output quality. However, the achievable speedup largely depends on the effectiveness of the drafting model. While model-based methods like EAGLE-2 are accurate but costly, retrieval-enhanced methods like SAM-Decoding rely on heuristic switching strategies that often trigger unnecessary retrievals. To address this, we propose ReSpec (\textbf{Re}trieval-enhanced \textbf{Spe}culative Decoding), a novel framework that transforms heuristic drafter switching into adaptive decision-making. ReSpec features three core innovations: 1) An \textbf{entropy-guided adaptive trigger} quantifies contextual predictability to initiate retrieval only when uncertainty is low, avoiding costly low-quality speculations. 2) A \textbf{feedback-driven candidate selection} leverages historical feedback to organize multiple high-quality candidates for parallel verification, maximizing retrieval utility. 3) A source-aware \textbf{relaxed verification strategy} applies strict checks to model-generated drafts while using a relaxed verification for retrieved drafts, achieving a better balance between accuracy and efficiency. Extensive experiments on Spec-Bench demonstrate that ReSpec achieves state-of-the-art acceleration,outperforming EAGLE-2 and SAM-Decoding by over $33\%$ and $25\%$, respectively, while maintaining output quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。