用参考谱图指导扩散模型,提升歌声合成音质与稳定性。
RDSinger: Reference-based Diffusion Network for Singing Voice Synthesis
- 以FastSpeech2生成的声谱为参考,引导扩散去噪过程。
- 在中高音转换区使用高斯模糊和加权损失,减少失真。
- 在中文歌曲数据集上超越现有最佳方法,适合音乐生成研究者。
歌声合成(SVS)旨在从乐谱生成高质量歌唱音频,需精确理解音符、音高和时长,不同于文本到语音任务。尽管扩散模型在图像、视频生成等任务中表现优异,但在SVS中受限于时间复杂度及对声学特征(尤其是音高过渡)的捕捉能力。现有方法虽利用先验分布并以压缩潜在状态作为扩散起点,但去噪步骤在全时长内质量提升不一致。本文提出RDSinger,一种基于参考的去噪扩散网络,用于生成高质量的歌声。受Animate Anyone启发,该模型以FastSpeech2生成的梅尔频谱作为参考,有效缓解去噪过程中的伪影。此外,针对音高过渡期间压缩潜在状态可能引入误导信息的问题,我们对部分参考频谱施加高斯模糊,并在这些区域调整损失权重。大量消融实验验证了方法的有效性。在中文歌曲数据集OpenCpop上的评估表明,RDSinger在性能上优于当前最先进方法。
原文摘要 · Abstract (English)
Singing voice synthesis (SVS) aims to produce high-fidelity singing audio from music scores, requiring a detailed understanding of notes, pitch, and duration, unlike text-to-speech tasks. Although diffusion models have shown exceptional performance in various generative tasks like image and video creation, their application in SVS is hindered by time complexity and the challenge of capturing acoustic features, particularly during pitch transitions. Some networks learn from the prior distribution and use the compressed latent state as a better start in the diffusion model, but the denoising step doesn't consistently improve quality over the entire duration. We introduce RDSinger, a reference-based denoising diffusion network that generates high-quality audio for SVS tasks. Our approach is inspired by Animate Anyone, a diffusion image network that maintains intricate appearance features from reference images. RDSinger utilizes FastSpeech2 mel-spectrogram as a reference to mitigate denoising step artifacts. Additionally, existing models could be influenced by misleading information on the compressed latent state during pitch transitions. We address this issue by applying Gaussian blur on partial reference mel-spectrogram and adjusting loss weights in these regions. Extensive ablation studies demonstrate the efficiency of our method. Evaluations on OpenCpop, a Chinese singing dataset, show that RDSinger outperforms current state-of-the-art SVS methods in performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。