端到端神经说话人分离框架,无需预先注册即可识别新说话人。
Sequence-to-Sequence Neural Diarization with Automatic Speaker Detection and Representation
- 基于序列到序列架构,联合学习说话人检测与表示
- 可自动发现新说话人并实时更新嵌入缓存,支持长音频分块处理
- 无需额外模型,适合在线/离线语音分离任务
本文提出一种新型序列到序列神经说话人分离(S2SND)框架,实现在线与离线说话人分离。该框架源于我们先前的目标说话人语音活动检测系统,通过解决两个关键问题演进为新的分离范式:1)说话人检测:可利用部分已知说话人嵌入,发现未知说话人并预测目标语音活动,无需提前进行说话人注册;2)说话人表示:可将预测的语音活动作为参考信息,从音频中同步提取说话人嵌入,且嵌入表示空间在整体分离网络中联合学习,不依赖额外说话人嵌入模型。推理时,S2SND框架可分块处理长音频。检测模块利用先前获得的说话人嵌入缓存,预测每个新音频块中已注册与未知说话人的语音活动;随后,根据表示模块的预测结果更新说话人嵌入缓存。假设每次小块滑动中最多出现一个新说话人,模型迭代预测每一块结果,并为后续块提取目标嵌入,直至信号结束。最后,最终的说话人嵌入缓存可对整段音频重新评分,实现高精度离线分离性能。
原文摘要 · Abstract (English)
This paper proposes a novel Sequence-to-Sequence Neural Diarization (S2SND) framework to perform online and offline speaker diarization. It is developed from the sequence-to-sequence architecture of our previous target-speaker voice activity detection system and then evolves into a new diarization paradigm by addressing two critical problems. 1) Speaker Detection: The proposed approach can utilize partially given speaker embeddings to discover the unknown speaker and predict the target voice activities in the audio signal. It does not require a prior diarization system for speaker enrollment in advance. 2) Speaker Representation: The proposed approach can adopt the predicted voice activities as reference information to extract speaker embeddings from the audio signal simultaneously. The representation space of speaker embedding is jointly learned within the whole diarization network without using an extra speaker embedding model. During inference, the S2SND framework can process long audio recordings blockwise. The detection module utilizes the previously obtained speaker-embedding buffer to predict both enrolled and unknown speakers' voice activities for each coming audio block. Next, the speaker-embedding buffer is updated according to the predictions of the representation module. Assuming that up to one new speaker may appear in a small block shift, our model iteratively predicts the results of each block and extracts target embeddings for the subsequent blocks until the signal ends. Finally, the last speaker-embedding buffer can re-score the entire audio, achieving highly accurate diarization performance as an offline system. Experimental results show that ...
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。