将大模型融入流式语音识别，提升准确率同时保持实时性。
Transducer-Llama: Integrating LLMs into Streamable Transducer-based Speech Recognition
- 用因子化转换器架构融合大语言模型，实现流式推理。
- 在LibriSpeech上相对基线降低17%词错误率，多语言数据上降32%。
- 通过弱转强语言模型替换策略，高效适配大模型并降低训练成本。
尽管大型语言模型（LLMs）已被应用于自动语音识别（ASR），但实现模型流式处理仍是挑战。本文提出一种新架构Transducer-Llama，将LLM整合进因子化转换器（FT）模型，天然支持流式识别。针对LLM大规模词汇带来的数据稀疏与训练成本高的问题，提出高效的词汇对齐技术。实验表明，直接用强预训练LLM预测器优化FT模型，仅带来有限提升；因此本文采用弱到强语言模型替换策略：先用弱语言模型进行RNN-T损失训练，再替换为强LLM，并使用最小词错误率（MWER）损失微调融合效果。在LibriSpeech及大规模多语言LibriSpeech数据集上的实验显示，所提流式Transducer-Llama方法相比强FT基线降低17%相对词错误率（WERR），相比RNN-T基线降低32% WERR。
原文摘要 · Abstract (English)
While large language models (LLMs) have been applied to automatic speech recognition (ASR), the task of making the model streamable remains a challenge. This paper proposes a novel model architecture, Transducer-Llama, that integrates LLMs into a Factorized Transducer (FT) model, naturally enabling streaming capabilities. Furthermore, given that the large vocabulary of LLMs can cause data sparsity issue and increased training costs for spoken language systems, this paper introduces an efficient vocabulary adaptation technique to align LLMs with speech system vocabularies. The results show that directly optimizing the FT model with a strong pre-trained LLM-based predictor using the RNN-T loss yields some but limited improvements over a smaller pre-trained LM predictor. Therefore, this paper proposes a weak-to-strong LM swap strategy, using a weak LM predictor during RNN-T loss training and then replacing it with a strong LLM. After LM replacement, the minimum word error rate (MWER) loss is employed to finetune the integration of the LLM predictor with the Transducer-Llama model. Experiments on the LibriSpeech and large-scale multi-lingual LibriSpeech corpora show that the proposed streaming Transducer-Llama approach gave a 17% relative WER reduction (WERR) over a strong FT baseline and a 32% WERR over an RNN-T baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。