用扩散模型改进语音识别,显著降低错误率。
Audio-Conditioned Diffusion LLMs for ASR and Deliberation Processing
- 将扩散语言模型作为语音识别后处理模块,利用双向注意力和去噪能力。
- 在LibriSpeech上测试集其他数据集错误率降至4.94%,相对提升12.3%。
- 音频条件嵌入至关重要,纯文本模型无法提升性能,适合语音处理研究者。
基于扩散的大语言模型(DLLMs)作为自回归解码器的替代方案正受到越来越多关注。本文针对自动语音识别(ASR)任务,对基于扩散的大型语言模型LLaDA进行了实证研究。首先将其作为外部反思式处理模块,用于Whisper-LLaMA的转录结果。通过利用LLaDA的双向注意力与去噪能力,探索了随机掩码、低置信度掩码及半自回归策略,结果显示,Whisper-LLaDA相比基线显著降低了词错误率(WER)。在LibriSpeech数据集上,最佳级联系统在test-clean/test-other上的WER分别为2.25%/4.94%,在test-other上相对于Whisper-LLaMA基线实现了12.3%的相对改进。相比之下,不使用声学特征的纯文本LLaDA未能提升准确率,凸显音频条件嵌入的重要性。进一步评估了Whisper-LLaDA作为独立解码器的性能,采用基于扩散和半自回归解码方式,多数配置推理速度优于Whisper-LLaMA基线,尽管识别精度略低。这些发现为扩散型大模型在语音识别中的应用提供了实证支持,并指明了优化方向。代码与模型已在https://github.com/liuzhan22/Diffusion-ASR开源。
原文摘要 · Abstract (English)
Diffusion-based large language models (DLLMs) have recently attracted growing interest as an alternative to autoregressive decoders. In this work, we present an empirical study on using the diffusion-based large language model LLaDA for automatic speech recognition (ASR). We first investigate its use as an external deliberation-based processing module for Whisper-LLaMA transcripts. By leveraging the bidirectional attention and denoising capabilities of LLaDA, we explore random masking, low-confidence masking, and semi-autoregressive strategies, showing that Whisper-LLaDA substantially reduces WER compared with the baseline. On LibriSpeech, the best cascade system achieves 2.25%/4.94% WER on test-clean/test-other, representing a 12.3% relative improvement over the Whisper-LLaMA baseline on the test-other split. In contrast, a plain-text LLaDA without acoustic features fails to improve accuracy, highlighting the importance of audio-conditioned embeddings. We further evaluate Whisper-LLaDA as a standalone decoder for ASR with diffusion-based and semi-autoregressive decoding. Most experimental configurations achieve faster inference than the Whisper-LLaMA baseline, although recognition accuracy is slightly lower. These findings offer an empirical view of diffusion-based LLMs for ASR and point to promising directions for improvements. Code and model are open-sourced at https://github.com/liuzhan22/Diffusion-ASR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。