实时处理长音频的说话人分离模型,支持最多8人、长达一小时的流式识别。
LS-EEND: Long-Form Streaming End-to-End Neural Diarization with Online Attractor Extraction
- 通过因果编码器与在线吸引子解码器,逐帧生成并更新说话人特征。
- 在多个数据集上达到新最优效果,如CALLHOME错误率12.11%。
- 适合长时间会议、广播等需低延迟实时处理的场景。
本文提出一种帧级在线/流式端到端神经说话人分离(EEND)方法,以帧对帧的方式检测说话人活动。模型主要由因果嵌入编码器和在线吸引子解码器构成。在基于自注意力的解码器中,说话人同时在时间和说话人维度建模,自动生成并更新新旧说话人的帧级吸引子。采用保留机制,特别适配长时对话,实现线性时间复杂度。提出多步渐进训练策略,从简单任务逐步过渡到多说话人和长音频的复杂任务。最终模型(称为长时流式EEND,LS-EEND)可实现高(最多8个)、灵活数量说话人及长达一小时音频的流式分离。实验表明:1)不依赖语音活动真值信息时,在所有数据集上均达到最新最优的在线说话人分离错误率,包括CALLHOME(12.11%)、DIHARD II(27.58%)、DIHARD III(19.61%)和AMI(20.76%);2)由于帧对帧处理方式与线性时间复杂度,相比现有在线模型,其实时因子降低数倍。
原文摘要 · Abstract (English)
This work proposes a frame-wise online/streaming end-to-end neural diarization (EEND) method, which detects speaker activities in a frame-in-frame-out fashion. The proposed model mainly consists of a causal embedding encoder and an online attractor decoder. Speakers are modeled in the self-attention-based decoder along both the time and speaker dimensions, and frame-wise speaker attractors are automatically generated and updated for new speakers and existing speakers, respectively. Retention mechanism is employed and especially adapted for long-form diarization with a linear temporal complexity. A multi-step progressive training strategy is proposed for gradually learning from easy tasks to hard tasks in terms of the number of speakers and audio length. Finally, the proposed model (referred to as long-form streaming EEND, LS-EEND) is able to perform streaming diarization for a high (up to 8) and flexible number speakers and very long (say one hour) audio recordings. Experiments on various simulated and real-world datasets show that: 1) when not using oracle speech activity information, the proposed model achieves new state-of-the-art online diarization error rate on all datasets, including CALLHOME (12.11%), DIHARD II (27.58%), DIHARD III (19.61%), and AMI (20.76%); 2) Due to the frame-in-frame-out processing fashion and the linear temporal complexity, the proposed model achieves several times lower real-time-factor than comparison online diarization models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。