用CTC提升生成序列相关性，让大模型推理更快更准。
Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration
- 用CTC建模生成过程中的序列依赖关系，增强候选文本质量。
- 实验表明接受率显著高于现有基线，推理速度更快。
- 适合追求高吞吐、低延迟的LLM部署场景。
大语言模型（LLM）推理加速在诸多应用场景中备受关注，其中推测解码（speculative decoding）展现出显著优势。该方法通过引入一个草稿模型辅助基础模型：草稿模型生成候选文本，基础模型验证其有效性并决定是否采纳。最终推理速度取决于草稿模型的解码速度和草稿被接受的比例。当前主流草稿模型通常以非自回归方式生成多个未来位置的令牌，但未考虑令牌间的相关性，导致解码速度快但接受率不理想。本文提出一种基于CTC的草稿模型，在生成阶段强化候选序列中令牌间的依赖关系，从而生成更高品质的候选序列。实验结果表明，相比强基线，所提方法可实现更高的接受率，进而获得更快的推理速度。
原文摘要 · Abstract (English)
Inference acceleration of large language models (LLMs) has been put forward in many application scenarios and speculative decoding has shown its advantage in addressing inference acceleration. Speculative decoding usually introduces a draft model to assist the base LLM where the draft model produces drafts and the base LLM verifies the draft for acceptance or rejection. In this framework, the final inference speed is decided by the decoding speed of the draft model and the acceptance rate of the draft provided by the draft model. Currently the widely used draft models usually generate draft tokens for the next several positions in a non-autoregressive way without considering the correlations between draft tokens. Therefore, it has a high decoding speed but an unsatisfactory acceptance rate. In this paper, we focus on how to improve the performance of the draft model and aim to accelerate inference via a high acceptance rate. To this end, we propose a CTC-based draft model which strengthens the correlations between draft tokens during the draft phase, thereby generating higher-quality draft candidate sequences. Experiment results show that compared to strong baselines, the proposed method can achieve a higher acceptance rate and hence a faster inference speed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。