用大模型语言知识提升语音识别效率与准确率
Boosting CTC-Based ASR Using LLM-Based Intermediate Loss Regularization
- 在编码器中间层引入语言模型损失，增强上下文建模
- 在三个数据集上降低词错误率，达到当前最优性能
- 保持无自回归解码速度优势，适合实时应用
端到端语音识别系统通过将所有组件整合到单一神经网络中革新了该领域，基于注意力的编码器-解码器模型达到了顶尖性能。然而，其自回归解码过程限制了推理速度，难以用于实时应用。相比之下，基于连接时序分类（CTC）的模型提供更快的非自回归解码，但难以有效建模语言依赖关系。为解决此问题，我们提出一种名为语言感知中间损失（LAIL）的新辅助损失框架，利用大语言模型（LLM）的语言知识增强基于CTC的语音识别。通过在编码器中间层添加连接层，LAIL将输出映射到LLM的嵌入空间，并在训练期间计算因果语言建模损失。该方法在保持CTC解码计算效率的同时提升了语言建模能力。使用Conformer架构和多个LLaMA模型，在LibriSpeech、TEDLIUM2和WSJ数据集上显著降低词错误率（WER），实现了基于CTC的语音识别的最先进性能，且计算开销极小。
原文摘要 · Abstract (English)
End-to-end (E2E) automatic speech recognition (ASR) systems have revolutionized the field by integrating all components into a single neural network, with attention-based encoder-decoder models achieving state-of-the-art performance. However, their autoregressive decoding process limits inference speed, making them unsuitable for real-time applications. In contrast, CTC-based models offer faster, non-autoregressive decoding but struggle to model linguistic dependencies effectively. Addressing this challenge, we propose a novel auxiliary loss framework called Language-Aware Intermediate Loss (LAIL) to enhance CTC-based ASR using the linguistic knowledge of large language models (LLMs). By attaching connector layers to intermediate encoder layers, LAIL maps outputs to the embedding space of an LLM and computes a causal language modeling loss during training. This approach enhances linguistic modeling while preserving the computational efficiency of CTC decoding. Using the Conformer architecture and various LLaMA models, we demonstrate significant improvements in Word Error Rate (WER) on the LibriSpeech, TEDLIUM2, and WSJ corpora, achieving state-of-the-art performance for CTC-based ASR with minimal computational overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。