对比不同模型组件对端到端语音分角色性能的影响,找到最优组合。
Dissecting the Segmentation Model of End-to-End Diarization with Vector Clustering
- 测试多种编码器、解码器、损失函数和分块大小的组合效果。
- 微调WavLM编码器+Conformer解码器在9个数据集上表现最佳。
- 新架构更适应长分块,显著提升整体性能,适合实际应用。
端到端语音分角色结合向量聚类是一种强大且实用的方法。尽管已有多种针对分割模型的改进,但其协同效应尚未充分评估。本文深入分析了主要架构选择对系统性能的影响。我们考察了不同编码器(SincNet、预训练及微调WavLM)、不同解码器(LSTM、Mamba、Conformer)、不同损失函数(多标签与多分类幂集)以及不同分块大小。在覆盖九个数据集的深入实验中发现,微调WavLM编码器始终显著优于其他方案。LSTM解码器被Mamba和Conformer超越;虽然Mamba对其他配置更具鲁棒性,但最佳性能仍来自使用Conformer编码器的架构。多标签与多分类幂集损失的错误分布不同,多分类损失几乎让所有模型表现更优,但在微调WavLM时,多标签损失反而更优。我们还评估了分块大小的影响,发现新架构更能处理长分块,可大幅提升系统性能。最佳系统在五个常用语音分角色数据集上达到当前最优结果。
原文摘要 · Abstract (English)
End-to-End Neural Diarization with Vector Clustering is a powerful and practical approach to perform Speaker Diarization. Multiple enhancements have been proposed for the segmentation model of these pipelines, but their synergy had not been thoroughly evaluated. In this work, we provide an in-depth analysis on the impact of major architecture choices on the performance of the pipeline. We investigate different encoders (SincNet, pretrained and finetuned WavLM), different decoders (LSTM, Mamba, and Conformer), different losses (multilabel and multiclass powerset), and different chunk sizes. Through in-depth experiments covering nine datasets, we found that the finetuned WavLM-based encoder always results in the best systems by a wide margin. The LSTM decoder is outclassed by Mamba- and Conformer-based decoders, and while we found Mamba more robust to other architecture choices, it is slightly inferior to our best architecture, which uses a Conformer encoder. We found that multilabel and multiclass powerset losses do not have the same distribution of errors. We confirmed that the multiclass loss helps almost all models attain superior performance, except when finetuning WavLM, in which case, multilabel is the superior choice. We also evaluated the impact of the chunk size on all aforementioned architecture choices and found that newer architectures tend to better handle long chunk sizes, which can greatly improve pipeline performance. Our best system achieved state-of-the-art results on five widely used speaker diarization datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。