arXiv:2601.19472cs.SD2026-01中稿 · ICASSP 2026被引 1

融合Conformer与Mamba优势,提升长音频说话人分割精度。

Dual-Strategy-Enhanced ConBiMamba for Neural Speaker Diarization

  • 用ExtBiMamba替代自注意力,降低内存开销并增强长序列建模。
  • 在6个数据集上4个达顶尖水平,尤其改善了换人点处的错误率。
  • 适合需要高效高精度语音分割的研究者与开发者使用。

Conformer与Mamba在语音建模中表现优异,但在说话人分割任务中存在局限:Mamba对局部细节和非线性模式捕捉不足;Conformer的自注意力机制在长语音序列中内存开销高,且长距离依赖建模易不稳定。这些缺陷严重影响分割任务对局部变化精确建模及跨长段的说话人一致性要求。为此,本文基于Pyannote流程提出双策略增强的ConBiMamba神经说话人分割系统。ConBiMamba结合Conformer与Mamba优势,利用其卷积与前馈结构提升局部特征提取能力,以ExtBiMamba替代自注意力,有效处理长音频同时缓解内存压力。为解决换人点附近错误率较高的问题,引入边界增强过渡损失(Boundary-Enhanced Transition Loss)以强化换人点检测。此外,提出分层特征聚合(Layer-wise Feature Aggregation)以更好利用多层表示。该系统在六个数据集上评估,四个达到当前最优性能。代码已开源:https://github.com/lz-hust/DSE-CBM。

原文摘要 · Abstract (English)

Conformer and Mamba have achieved strong performance in speech modeling but face limitations in speaker diarization. Mamba is efficient but struggles with local details and nonlinear patterns. Conformer's self-attention incurs high memory overhead for long speech sequences and may cause instability in long-range dependency modeling. These limitations are critical for diarization, which requires both precise modeling of local variations and robust speaker consistency over extended spans. To address these challenges, we first apply ConBiMamba for speaker diarization. We follow the Pyannote pipeline and propose the Dual-Strategy-Enhanced ConBiMamba neural speaker diarization system. ConBiMamba integrates the strengths of Conformer and Mamba, where Conformer's convolutional and feed-forward structures are utilized to improve local feature extraction. By replacing Conformer's self-attention with ExtBiMamba, ConBiMamba efficiently handles long audio sequences while alleviating the high memory cost of self-attention. Furthermore, to address the problem of the higher DER around speaker change points, we introduce the Boundary-Enhanced Transition Loss to enhance the detection of speaker change points. We also propose Layer-wise Feature Aggregation to enhance the utilization of multi-layer representations. The system is evaluated on six diarization datasets and achieves state-of-the-art performance on four of them. The source code of our study is available at https://github.com/lz-hust/DSE-CBM.

说话人分割Mamba语音建模深度学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。