根据模型置信度动态调整生成长度，提升大模型推理速度与稳定性。
Confidence-Modulated Speculative Decoding for Large Language Models
- 基于熵和置信度动态调节每轮推测生成的词元数量
- 在机器翻译和摘要任务中实现更快推理，且保持或提升评分
- 适用于不确定性强、输入复杂的场景，适合实际部署
推测解码通过先生成后验证的并行机制显著加速自回归推理。然而，现有方法依赖固定生成长度和刚性验证标准，在不同模型不确定性与输入复杂度下适应性差。本文提出一种基于置信度调制的推测解码框架，利用熵和置信区间度量生成器输出分布的不确定性，动态调整每轮推测生成的词元数。该机制有效降低回滚频率，提升资源利用率并维持输出质量。同时，验证过程也根据相同置信信号进行调制，实现更灵活的接受策略而不牺牲生成质量。在机器翻译和摘要任务上的实验表明，该方法相比标准推测解码有显著加速，同时保持或提升BLEU和ROUGE分数。所提方法为大语言模型在不确定环境下的高效鲁棒解码提供了一个可插拔的原理性方案。
原文摘要 · Abstract (English)
Speculative decoding has emerged as an effective approach for accelerating autoregressive inference by parallelizing token generation through a draft-then-verify paradigm. However, existing methods rely on static drafting lengths and rigid verification criteria, limiting their adaptability across varying model uncertainties and input complexities. This paper proposes an information-theoretic framework for speculative decoding based on confidence-modulated drafting. By leveraging entropy and margin-based uncertainty measures over the drafter's output distribution, the proposed method dynamically adjusts the number of speculatively generated tokens at each iteration. This adaptive mechanism reduces rollback frequency, improves resource utilization, and maintains output fidelity. Additionally, the verification process is modulated using the same confidence signals, enabling more flexible acceptance of drafted tokens without sacrificing generation quality. Experiments on machine translation and summarization tasks demonstrate significant speedups over standard speculative decoding while preserving or improving BLEU and ROUGE scores. The proposed approach offers a principled, plug-in method for efficient and robust decoding in large language models under varying conditions of uncertainty.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。