将视觉特征同时用于编码器和解码器，显著提升语音识别在噪声下的表现。
Noise-Robust AV-ASR Using Visual Features Both in the Whisper Encoder and Decoder
- 视觉特征同时接入编码器与解码器，增强音视频交互与模态权重调节。
- 在0dB信噪比下，小模型相对改进35%，中模型达57%（WER从9.53%降至4.07%）。
- 适用于需要高噪声鲁棒性的音视频语音识别场景，如智能设备、会议系统。
在音频视觉自动语音识别（AV-ASR）系统中，利用预训练语音识别模型融合视觉信息已被证明是提升噪声鲁棒性的有效方法。本文基于Whisper ASR，提出一种简单高效的视觉融合策略——在编码器和解码器中同时使用视觉特征（双用法），以在编码器中学习音视频交互，在解码器中动态加权模态。进一步对比了不同规模Whisper模型的视觉融合效果。所提双用法在各类噪声下均表现出一致的鲁棒性提升：在0dB信噪比的白噪音环境下，Whisper small模型的相对误差降低35%（WER：4.41% vs. 6.83%），Whisper medium模型降低57%（WER：4.07% vs. 9.53%）。通过消融实验分析模块设计与融合方式的影响。在1929小时音视频数据上微调后，该方法在LRS3 AV-ASR基准测试中，于MUSAN白噪音环境平均达到4.08% WER，NoiseX环境为4.43%（跨多种信噪比），创下新纪录。代码已开源。
原文摘要 · Abstract (English)
In audiovisual automatic speech recognition (AV-ASR) systems, information fusion of visual features in a pre-trained ASR has been proven as a promising method to improve noise robustness. In this work, based on the prominent Whisper ASR, first, we propose a simple and effective visual fusion method -- use of visual features both in encoder and decoder (dual-use) -- to learn the audiovisual interactions in the encoder and to weigh modalities in the decoder. Second, we compare visual fusion methods in Whisper models of various sizes. Our proposed dual-use method shows consistent noise robustness improvement, e.g., a 35% relative improvement (WER: 4.41% vs. 6.83%) based on Whisper small, and a 57% relative improvement (WER: 4.07% vs. 9.53%) based on Whisper medium, compared to typical reference middle fusion in babble noise with a signal-to-noise ratio (SNR) of 0dB. Third, we conduct ablation studies examining the impact of various module designs and fusion options. Fine-tuned on 1929 hours of audiovisual data, our dual-use method using Whisper medium achieves 4.08% (MUSAN babble noise) and 4.43% (NoiseX babble noise) average WER across various SNRs, thereby establishing a new state-of-the-art in noisy conditions on the LRS3 AV-ASR benchmark. Our code is at https://github.com/ifnspaml/Dual-Use-AVASR
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。