用环境视觉线索增强语音，提升嘈杂场景下听清对话能力
SAV-SE: Scene-aware Audio-Visual Speech Enhancement with Selective State Space Model
- 融合面部与环境视觉信息，构建上下文感知的语音增强模型
- 在MUSIC、AVSpeech等数据集上，语音清晰度提升2.1~3.8dB
- 适合需要高鲁棒性语音处理的智能会议、车载系统等场景
语音增强在诸多应用中至关重要，视觉信息的引入已被证明能带来显著优势。然而，现有研究多聚焦于人脸和嘴唇运动，这些特征在遮挡或远距离拍摄时可能失效。本文提出新任务SAV-SE，首次利用同步视频中的丰富环境上下文作为辅助线索来识别噪声类型，从而提升语音增强效果。我们提出VC-S²E方法，结合Conformer与Mamba模块的优势以实现互补。在公开数据集MUSIC、AVSpeech和AudioSet上进行大量实验，结果表明该方法优于现有主流方法。源代码将公开。项目演示页：https://AVSEPage.github.io/
原文摘要 · Abstract (English)
Speech enhancement plays an essential role in various applications, and the integration of visual information has been demonstrated to bring substantial advantages. However, the majority of current research concentrates on the examination of facial and lip movements, which can be compromised or entirely inaccessible in scenarios where occlusions occur or when the camera view is distant. Whereas contextual visual cues from the surrounding environment have been overlooked: for example, when we see a dog bark, our brain has the innate ability to discern and filter out the barking noise. To this end, in this paper, we introduce a novel task, i.e. SAV-SE. To our best knowledge, this is the first proposal to use rich contextual information from synchronized video as auxiliary cues to indicate the type of noise, which eventually improves the speech enhancement performance. Specifically, we propose the VC-S$^2$E method, which incorporates the Conformer and Mamba modules for their complementary strengths. Extensive experiments are conducted on public MUSIC, AVSpeech and AudioSet datasets, where the results demonstrate the superiority of VC-S$^2$E over other competitive methods. We will make the source code publicly available. Project demo page: https://AVSEPage.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。