用短片段训练的模型，能处理超长语音且保持说话人关联。
Multi-Utterance Speech Separation and Association Trained on Short Segments
- 设计频时递归网络，分频带建模时间、全频段建模频率依赖。
- 在10秒短段训练后，可处理21-121秒长语音，跨语段说话人一致。
- 无需分段拼接，轻量模型（0.9M参数）直接推理，部署更简单。
当前基于深度神经网络的语音分离面临根本挑战：模型受限于计算资源需在短片段上训练，但实际应用常需处理远超训练长度的长录音，且包含多轮发言。本文研究现有方法在此场景下的表现，并提出一种频时递归神经网络（FTRNN），有效弥合该差距。FTRNN采用全频段模块建模每帧内的频率依赖关系，子频带模块建模各频带内的时序模式。尽管仅在10秒固定长度片段上训练，该模型在处理21至121秒的长音频时仍表现稳健，且能保持跨语段间隙（超过训练中出现的情况）的说话人关联。与传统‘分段分离再拼接’方式不同，本方法以轻量级架构（0.9百万参数）直接对长音频进行端到端推理，避免分段边界失真，简化部署流程。实验表明，FTRNN在多轮语音分离与说话人关联任务中具备优异泛化能力。
原文摘要 · Abstract (English)
Current deep neural network (DNN) based speech separation faces a fundamental challenge -- while the models need to be trained on short segments due to computational constraints, real-world applications typically require processing significantly longer recordings with multiple utterances per speaker than seen during training. In this paper, we investigate how existing approaches perform in this challenging scenario and propose a frequency-temporal recurrent neural network (FTRNN) that effectively bridges this gap. Our FTRNN employs a full-band module to model frequency dependencies within each time frame and a sub-band module that models temporal patterns in each frequency band. Despite being trained on short fixed-length segments of 10 s, our model demonstrates robust separation when processing signals significantly longer than training segments (21-121 s) and preserves speaker association across utterance gaps exceeding those seen during training. Unlike the conventional segment-separation-stitch paradigm, our lightweight approach (0.9 M parameters) performs inference on long audio without segmentation, eliminating segment boundary distortions while simplifying deployment. Experimental results demonstrate the generalization ability of FTRNN for multi-utterance speech separation and speaker association.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。