延迟融合让大模型更高效地参与语音识别解码
Delayed Fusion: Integrating Large Language Models into First-Pass Decoding in End-to-end Speech Recognition
- 解码时延迟调用大模型，只对少数候选结果评分
- 在LibriHeavy上速度提升30%，错误率降低12%
- 无需重训练，直接复用现有大模型，适合实际部署
本文提出一种高效的端到端语音识别（E2E-ASR）解码方法——延迟融合，用于整合大型语言模型（LLM）。尽管浅层融合是主流方式，但大模型推理成本高且存在词汇不匹配问题。为解决此问题，需重训练模型，往往耗时甚至不可行。延迟融合在解码过程中延迟应用大模型打分，仅对少数候选序列进行评分，显著减少大模型调用次数。该方法还能在不同分词方式下重新对齐识别结果。实验表明，在LibriHeavy数据集和OpenLLaMA 3B、7B、Mistral 7B三个公开大模型上，相比浅层融合与N-best重评分，延迟融合在保持准确率的同时提升了30%的解码速度。
原文摘要 · Abstract (English)
This paper presents an efficient decoding approach for end-to-end automatic speech recognition (E2E-ASR) with large language models (LLMs). Although shallow fusion is the most common approach to incorporate language models into E2E-ASR decoding, we face two practical problems with LLMs. (1) LLM inference is computationally costly. (2) There may be a vocabulary mismatch between the ASR model and the LLM. To resolve this mismatch, we need to retrain the ASR model and/or the LLM, which is at best time-consuming and in many cases not feasible. We propose "delayed fusion," which applies LLM scores to ASR hypotheses with a delay during decoding and enables easier use of pre-trained LLMs in ASR tasks. This method can reduce not only the number of hypotheses scored by the LLM but also the number of LLM inference calls. It also allows re-tokenizion of ASR hypotheses during decoding if ASR and LLM employ different tokenizations. We demonstrate that delayed fusion provides improved decoding speed and accuracy compared to shallow fusion and N-best rescoring using the LibriHeavy ASR corpus and three public LLMs, OpenLLaMA 3B & 7B and Mistral 7B.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。