用双上下文学习降低语音增强模型复杂度,提升泛化能力。
Conditional Latent Diffusion-Based Speech Enhancement Via Dual Context Learning
- 在低维潜空间中结合条件扩散模型与双上下文学习
- 仅需较少迭代步数即超越现有方法,对陌生噪声泛化更强
- 适合需要快速推理和强鲁棒性的语音增强场景
近期,扩散概率模型通过生成方法推动了语音增强的发展。然而,现有基于扩散的方法主要在高维波形或频谱域进行生成,导致生成复杂度高、推理速度慢。同时,这些方法多聚焦于干净语音分布建模,对噪声分布探索不足,限制了扩散模型在语音增强中的判别能力。为解决上述问题,我们提出一种新方法:将条件潜变量扩散模型(cLDM)与双上下文学习(DCL)结合。该方法利用变分自编码器(VAE)将梅尔频谱图压缩至低维潜空间,再通过DCL过程将干净语音与背景噪声的潜表示分别转化为高斯噪声,并训练一个参数化模型以噪声潜表示和文本嵌入为条件反向重构。由于在低维空间操作,潜表示降低了生成复杂度;双上下文学习增强了模型对多样化及未见噪声环境的处理能力。实验表明,所提方法在较少迭代步数下性能优于现有扩散方法,并展现出对域外噪声数据集的优异泛化能力(https://github.com/modelscope/ClearerVoice-Studio)。
原文摘要 · Abstract (English)
Recently, the application of diffusion probabilistic models has advanced speech enhancement through generative approaches. However, existing diffusion-based methods have focused on the generation process in high-dimensional waveform or spectral domains, leading to increased generation complexity and slower inference speeds. Additionally, these methods have primarily modelled clean speech distributions, with limited exploration of noise distributions, thereby constraining the discriminative capability of diffusion models for speech enhancement. To address these issues, we propose a novel approach that integrates a conditional latent diffusion model (cLDM) with dual-context learning (DCL). Our method utilizes a variational autoencoder (VAE) to compress mel-spectrograms into a low-dimensional latent space. We then apply cLDM to transform the latent representations of both clean speech and background noise into Gaussian noise by the DCL process, and a parameterized model is trained to reverse this process, conditioned on noisy latent representations and text embeddings. By operating in a lower-dimensional space, the latent representations reduce the complexity of the generation process, while the DCL process enhances the model's ability to handle diverse and unseen noise environments. Our experiments demonstrate the strong performance of the proposed approach compared to existing diffusion-based methods, even with fewer iterative steps, and highlight the superior generalization capability of our models to out-of-domain noise datasets (https://github.com/modelscope/ClearerVoice-Studio).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。