让预训练语音模型学会利用多麦克风空间信息，提升说话人分离效率
Spatially Aware Self-Supervised Models for Multi-Channel Neural Speaker Diarization
- 在早期层插入通道通信模块，使WavLM具备空间感知能力
- 多通道语音嵌入通过空间注意力融合，在5个数据集上均超越单通道基线
- 无需依赖复杂后处理，适合实际部署的多麦克风语音系统
自监督模型如WavLM在神经说话人分离任务中表现优异，但其通常在单通道录音上预训练，限制了在多通道场景下的效果。现有系统常使用DOVER-Lap融合各通道输出，虽有效但计算开销大，且未充分利用空间信息。本文基于DiariZen框架，在其基础上引入轻量级通道通信模块，使预训练的WavLM具备空间感知能力，该方法对麦克风数量和阵列拓扑均无依赖，适用性广。进一步提出基于空间注意力权重的多通道说话人嵌入融合机制。在五个公开数据集上的实验表明，该方法相比单通道基线有持续提升，性能与效率均优于DOVER-Lap。代码已开源。
原文摘要 · Abstract (English)
Self-supervised models such as WavLM have demonstrated strong performance for neural speaker diarization. However, these models are typically pre-trained on single-channel recordings, limiting their effectiveness in multi-channel scenarios. Existing diarization systems built on these models often rely on DOVER-Lap to combine outputs from individual channels. Although effective, this approach incurs substantial computational overhead and fails to fully exploit spatial information. In this work, building on DiariZen, a pipeline that combines WavLM-based local endto-end neural diarization with speaker embedding clustering, we introduce a lightweight approach to make pre-trained WavLM spatially aware by inserting channel communication modules into the early layers. Our method is agnostic to both the number of microphone channels and array topologies, ensuring broad applicability. We further propose to fuse multi-channel speaker embeddings by leveraging spatial attention weights. Evaluations on five public datasets show consistent improvements over single-channel baselines and demonstrate superior performance and efficiency compared with DOVER-Lap. Our source code is publicly available at https://github.com/BUTSpeechFIT/DiariZen.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。