用扩散模型+非负矩阵分解，无监督提升音视频语音增强效果
Diffusion-based Unsupervised Audio-visual Speech Enhancement
- 扩散模型结合视频信息预训练，生成清晰语音分布
- 迭代优化噪声参数，使语音增强性能超越纯音频方法
- 推理速度更快，适合实时应用，无需标注数据
本文提出一种新的无监督音视频语音增强（AVSE）方法，将基于扩散的音视频语音生成模型与基于非负矩阵分解（NMF）的噪声模型相结合。首先，利用对应视频数据对扩散模型进行清洁语音的预训练，以模拟语音生成分布。随后，该预训练模型与NMF噪声模型协同，通过迭代方式估计干净语音。具体地，在反向扩散过程中引入基于扩散的后验采样策略，每轮迭代后得到语音估计，并用于更新噪声参数。实验结果表明，所提方法不仅优于纯音频的对比方法，且在泛化能力上超过近期的有监督生成式AVSE方法。此外，新提出的推理算法在推理速度与性能之间取得更好平衡，相较于以往扩散基方法更具实用性。代码与演示见：https://jeaneudesayilo.github.io/fast_UdiffSE
原文摘要 · Abstract (English)
This paper proposes a new unsupervised audio-visual speech enhancement (AVSE) approach that combines a diffusion-based audio-visual speech generative model with a non-negative matrix factorization (NMF) noise model. First, the diffusion model is pre-trained on clean speech conditioned on corresponding video data to simulate the speech generative distribution. This pre-trained model is then paired with the NMF-based noise model to estimate clean speech iteratively. Specifically, a diffusion-based posterior sampling approach is implemented within the reverse diffusion process, where after each iteration, a speech estimate is obtained and used to update the noise parameters. Experimental results confirm that the proposed AVSE approach not only outperforms its audio-only counterpart but also generalizes better than a recent supervised-generative AVSE method. Additionally, the new inference algorithm offers a better balance between inference speed and performance compared to the previous diffusion-based method. Code and demo available at: https://jeaneudesayilo.github.io/fast_UdiffSE
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。