用CTC编码器做草稿，加速语音识别并降低错误率。
Self-Speculative Decoding for LLM-based ASR with CTC Encoder Drafts
- 用CTC输出熵判断是否直接采纳，否则用大模型快速验证
- 在九个语料库上实现4.4倍加速，WER降至5.58%
- 适合追求高效高精度语音识别的开发者使用
我们提出自洽推测解码方法，利用CTC编码器作为草稿模型，加速语音感知大语言模型的自回归推理并提升语音识别准确率。该方法分三步：(1) 若CTC输出分布的帧熵低于阈值，则直接接受贪婪解码结果；(2) 否则以宽松的词元似然标准，在一次大模型前向传播中验证草稿；(3) 验证失败则从已接受的前缀恢复自回归解码。在九个语料库、五种语言上的实验表明，该方法可同时加速推理并降低词错误率（WER）。在HuggingFace Open ASR基准上，使用10亿参数大模型与4.4亿参数CTC编码器，达到5.58%的记录级WER，逆实时因子提升4.4倍，仅比自回归搜索高出12%相对WER。代码与模型权重以宽松许可公开。
原文摘要 · Abstract (English)
We propose self-speculative decoding for speech-aware LLMs by using the CTC encoder as a draft model to accelerate auto-regressive (AR) inference and improve ASR accuracy. Our three-step procedure works as follows: (1) if the frame entropies of the CTC output distributions are below a threshold, the greedy CTC hypothesis is accepted as final; (2) otherwise, the CTC hypothesis is verified in a single LLM forward pass using a relaxed acceptance criterion based on token likelihoods; (3) if verification fails, AR decoding resumes from the accepted CTC prefix. Experiments on nine corpora and five languages show that this approach can simultaneously accelerate decoding and reduce WER. On the HuggingFace Open ASR benchmark with a 1B parameter LLM and 440M parameter CTC encoder, we achieve a record 5.58% WER and improve the inverse real time factor by a factor of 4.4 with only a 12% relative WER increase over AR search. Code and model weights are publicly available under a permissive license.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。