通过分阶段生成流程提升语音分离的清晰度与自然度
SoloSpeech: Enhancing Intelligibility and Quality in Target Speech Extraction through a Cascaded Generative Pipeline
- 采用无说话人嵌入的级联生成架构，利用提示音的潜在空间对齐
- 在Libri2Mix上达到最新语音可懂度与音质水平
- 对真实场景和域外数据泛化能力强，适合实际部署
目标语音提取（TSE）旨在通过说话人特定线索（通常为辅助音频）从多人混叠中分离出目标说话人的声音。尽管近期基于判别式模型的方法在听觉质量上表现优异，但常引入不自然的伪影，且对训练与测试环境差异敏感。而生成式模型在可懂度与音质方面仍显不足。为此，我们提出SoloSpeech，一种融合压缩、提取、重建与修正的级联生成管道。该方法采用无需说话人嵌入的目标提取器，利用提示音频的潜在空间条件信息，与混合音频潜在空间对齐，避免特征失配。在广泛使用的Libri2Mix数据集上的评估显示，SoloSpeech在目标语音提取的可懂度与音质上达到新基准，并在域外数据及真实场景中展现出卓越泛化能力。
原文摘要 · Abstract (English)
Target Speech Extraction (TSE) aims to isolate a target speaker's voice from a mixture of multiple speakers by leveraging speaker-specific cues, typically provided as auxiliary audio (a.k.a. cue audio). Although recent advancements in TSE have primarily employed discriminative models that offer high perceptual quality, these models often introduce unwanted artifacts, reduce naturalness, and are sensitive to discrepancies between training and testing environments. On the other hand, generative models for TSE lag in perceptual quality and intelligibility. To address these challenges, we present SoloSpeech, a novel cascaded generative pipeline that integrates compression, extraction, reconstruction, and correction processes. SoloSpeech features a speaker-embedding-free target extractor that utilizes conditional information from the cue audio's latent space, aligning it with the mixture audio's latent space to prevent mismatches. Evaluated on the widely-used Libri2Mix dataset, SoloSpeech achieves the new state-of-the-art intelligibility and quality in target speech extraction while demonstrating exceptional generalization on out-of-domain data and real-world scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。