arXiv:2506.00466eess.AScs.SD2025-06IJCAI被引 12

通过多尺度对齐提升脑电与语音的同步性,增强目标说话人提取效果。

M3ANet: Multi-scale and Multi-Modal Alignment Network for Brain-Assisted Target Speaker Extraction

  • 用对比学习对齐脑电与语音的时间特征,解决模态不同步问题。
  • 采用多尺度卷积与GroupMamba模块编码语音,捕获深层时序信息。
  • 在三个公开数据集上超越现有方法,适合脑机接口和语音分离研究者。

脑辅助目标说话人提取(TSE)旨在利用脑神经活动(如脑电图,EEG)从混合语音中提取关注的语音。然而,现有模型忽略了语音与EEG模态间的时间错位问题,影响了提取性能。此外,当前模型的语音编码器通常仅使用基础的时序操作(如一维卷积),难以有效提取目标说话人信息。为此,本文提出多尺度多模态对齐网络(M3ANet)。为消除语音与EEG在时间上的不一致,设计了基于对比学习的模态对齐模块,实现两模态特征的时间对齐。同时,引入结合多尺度卷积与GroupMamba模块的语音编码器,从不同方向扫描各尺度语音特征,从而更充分地捕捉深度序列信息。在三个公开数据集上的实验结果表明,所提模型在多个评估指标上均优于现有最先进方法,验证了其有效性。源代码已公开:https://github.com/fchest/M3ANet。

原文摘要 · Abstract (English)

The brain-assisted target speaker extraction (TSE) aims to extract the attended speech from mixed speech by utilizing the brain neural activities, for example Electroencephalography (EEG). However, existing models overlook the issue of temporal misalignment between speech and EEG modalities, which hampers TSE performance. In addition, the speech encoder in current models typically uses basic temporal operations (e.g., one-dimensional convolution), which are unable to effectively extract target speaker information. To address these issues, this paper proposes a multi-scale and multi-modal alignment network (M3ANet) for brain-assisted TSE. Specifically, to eliminate the temporal inconsistency between EEG and speech modalities, the modal alignment module that uses a contrastive learning strategy is applied to align the temporal features of both modalities. Additionally, to fully extract speech information, multi-scale convolutions with GroupMamba modules are used as the speech encoder, which scans speech features at each scale from different directions, enabling the model to capture deep sequence information. Experimental results on three publicly available datasets show that the proposed model outperforms current state-of-the-art methods across various evaluation metrics, highlighting the effectiveness of our proposed method. The source code is available at: https://github.com/fchest/M3ANet.

脑机接口语音分离多模态对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。