arXiv:2510.05295cs.SDcs.AI2025-10

融合音视频信息提升语音增强效果,实现更清晰的说话人声音还原。

AUREXA-SE: Audio-Visual Unified Representation Exchange Architecture with Cross-Attention and Squeezeformer for Speech Enhancement

  • 采用双向交叉注意力机制实现音视频特征深度融合。
  • 在真实噪声环境下,语音可懂度(STOI)达0.516,质量(PESQ)达1.323。
  • 适合需要高保真语音重建的会议系统、智能设备等场景使用。

本文提出AUREXA-SE(基于交叉注意力与Squeezeformer的音视频统一表征交换架构),一种面向音视频语音增强(AVSE)的渐进式双模态框架。该模型通过基于U-Net的1D卷积编码器处理原始音频波形,并利用Swin Transformer V2高效提取视觉特征。核心是新型双向交叉注意力机制,促进模态间深层上下文融合,实现丰富互补的表征学习。为捕捉融合嵌入中的时序依赖性,引入轻量级Squeezeformer块,结合卷积与注意力模块。增强后的嵌入通过类似U-Net的解码器直接重构波形,确保听觉一致且可懂。实验表明,相较于噪声基线,性能显著提升:STOI达0.516,PESQ为1.323,SI-SDR为-4.322 dB。源代码已开源于https://github.com/mtanveer1/AVSEC-4-Challenge-2025。

原文摘要 · Abstract (English)

In this paper, we propose AUREXA-SE (Audio-Visual Unified Representation Exchange Architecture with Cross-Attention and Squeezeformer for Speech Enhancement), a progressive bimodal framework tailored for audio-visual speech enhancement (AVSE). AUREXA-SE jointly leverages raw audio waveforms and visual cues by employing a U-Net-based 1D convolutional encoder for audio and a Swin Transformer V2 for efficient and expressive visual feature extraction. Central to the architecture is a novel bidirectional cross-attention mechanism, which facilitates deep contextual fusion between modalities, enabling rich and complementary representation learning. To capture temporal dependencies within the fused embeddings, a stack of lightweight Squeezeformer blocks combining convolutional and attention modules is introduced. The enhanced embeddings are then decoded via a U-Net-style decoder for direct waveform reconstruction, ensuring perceptually consistent and intelligible speech output. Experimental evaluations demonstrate the effectiveness of AUREXA-SE, achieving significant performance improvements over noisy baselines, with STOI of 0.516, PESQ of 1.323, and SI-SDR of -4.322 dB. The source code of AUREXA-SE is available at https://github.com/mtanveer1/AVSEC-4-Challenge-2025.

语音增强音视频融合交叉注意力Squeezeformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。