将Mamba与共享时空注意力结合,提升语音增强的泛化能力。
MambAttention: Mamba with Multi-Head Attention for Generalizable Single-Channel Speech Enhancement
- 融合Mamba与共享时空多头注意力,构建新混合架构
- 在两个域外数据集上超越现有主流模型性能
- 适用于需要跨数据集泛化的语音增强场景
随着Mamba和xLSTM等序列模型的兴起,多项研究已证明其在单通道语音增强和音频表征学习中达到或超越当前最优水平。然而,已有研究表明,LSTM和Mamba等序列模型易对训练集过拟合。此前工作显示,在LSTM中加入自注意力可显著提升单通道语音增强的泛化性能。但混合Mamba与时频多头注意力模型及其泛化性能尚未被探索。本文提出一种新型混合架构MambAttention,结合Mamba与共享时-频多头注意力模块,实现可泛化的单通道语音增强。为训练模型,我们引入VB-DemandEx数据集,该数据集基于VoiceBank+Demand但包含更复杂噪声类型和更低信噪比。在VB-DemandEx上训练后,MambAttention在两个域外数据集(DNS 2020无混响、EARS-WHAM_v2)上所有报告指标均显著优于同等复杂度的LSTM、xLSTM、Mamba及Conformer基线系统。其泛化性能与生成式扩散模型相当,且媲美语言模型基线。消融实验表明,时-频多头注意力模块间权重共享对泛化至关重要。此外,将共享时-频多头注意力模块集成至LSTM和xLSTM也带来显著性能提升,但整体仍逊于MambAttention在跨语料库泛化上的表现。
原文摘要 · Abstract (English)
With new sequence models like Mamba and xLSTM, several studies have shown that these models match or outperform the state-of-the-art in single-channel speech enhancement and audio representation learning. However, prior research has demonstrated that sequence models like LSTM and Mamba tend to overfit to the training set. To address this, previous works have shown that adding self-attention to LSTMs substantially improves generalization performance for single-channel speech enhancement. Nevertheless, neither the concept of hybrid Mamba and time-frequency attention models nor their generalization performance have been explored for speech enhancement. In this paper, we propose a novel hybrid architecture, MambAttention, which combines Mamba and shared time- and frequency-multi-head attention modules for generalizable single-channel speech enhancement. To train our model, we introduce VB-DemandEx, a dataset inspired by VoiceBank+Demand but with more challenging noise types and lower signal-to-noise ratios. Trained on VB-DemandEx, MambAttention significantly outperforms existing state-of-the-art discriminative LSTM-, xLSTM-, Mamba-, and Conformer-based systems of similar complexity across all reported metrics on two out-of-domain datasets: DNS 2020 without reverberation and EARS-WHAM_v2. MambAttention also matches or outperforms generative diffusion models in generalization performance while being competitive with language model baselines. Ablation studies highlight the importance of weight sharing between time- and frequency-multi-head attention modules for generalization performance. Finally, we explore integrating the shared time- and frequency-multi-head attention modules with LSTM and xLSTM, which yields a notable performance improvement on the out-of-domain datasets. Yet, MambAttention remains superior for cross-corpus generalization across all reported evaluation metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。