用信息瓶颈提升大模型在噪声下的音视频语音识别能力
VIB-AVSR: Variational Information Bottleneck for Noise-Robust LLM-Based Audio-Visual Speech Recognition
- 在大模型中插入变分信息瓶颈层，约束噪声下的特征表示
- 多种信噪比和噪声类型下性能下降减少，最高提升12.3%
- 无需修改结构或额外数据，可直接适配现有模型
音视频语音识别同时利用音频和视觉信号，其中唇动视觉信息能在音频噪声环境下辅助识别。近年来，基于大语言模型（LLM）的音视频语音识别模型通过连接预训练的音视频编码器与大语言模型，已在纯净条件下取得优异表现。然而，这些模型主要针对干净音频优化，缺乏对大模型主干在噪声下鲁棒性的关注。现有方法未显式构建噪声下稳定表征的机制，导致在噪声环境中性能显著下降。为此，我们提出VIB-AVSR，将变分信息瓶颈（Variational Information Bottleneck, VIB）层嵌入到大模型主干的特定位置，以正则化特征表示。实验表明，VIB-AVSR在多个信噪比（SNR）水平和不同噪声类型下均有效缓解性能退化，且无需架构修改或额外训练数据。
原文摘要 · Abstract (English)
Audio-Visual Speech Recognition takes two input modalities, acoustic and visual streams, where visual information from lip movements aids recognition when audio is noisy. Recently, LLM-based AVSR models have emerged as a promising paradigm by connecting pre-trained audio-visual encoders to an LLM, achieving strong results in clean conditions. However, these models are predominantly optimized for clean acoustic conditions, with limited attention to making the LLM backbone robust to noise. No explicit mechanism is employed to produce stable representations under corrupted audio, leading to performance degradation in noisy environments. To address this, we propose VIB-AVSR, which integrates Variational Information Bottleneck layers at targeted positions within the LLM backbone to regularize representations. VIB-AVSR reduces degradation under noisy conditions across multiple SNR levels and noise types, without requiring architectural modifications or additional training data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。