让机器唱出非人类声音,打通创意场景的歌声生成难题
CartoonSing: Unifying Human and Nonhuman Timbres in Singing Generation
- 统一框架整合歌声合成与转换,支持人类和非人类音色
- 在无标注非人声数据下仍能生成符合音乐性的非人声歌声
- 适合游戏、动画等需要奇幻角色歌声的创作者使用
歌声合成(SVS)和歌声转换(SVC)在生成自然人声方面已取得显著进展,但现有系统仅限于人类音色,难以生成超出人类范围的声音,而这类需求在游戏、电影和虚拟角色中日益增长。本文提出非人类歌声生成(NHSG),涵盖非人类歌声合成(NHSVS)和非人类歌声转换(NHSVC),作为一项新的机器学习任务,旨在生成具有非人类音色特征且音乐连贯的歌声。该任务因非人声数据稀缺、符号对齐缺失以及人与非人声间音色差距大而极具挑战。为此,我们提出 CartoonSing 框架,采用两阶段流程:第一阶段使用标注的人声数据训练音符表示编码器;第二阶段通过音色感知声码器重建人声与非人声波形。实验表明,CartoonSing 能成功生成非人声歌声,可泛化至新音色,并将传统 SVS 与 SVC 扩展至创意类非人声歌声生成。
原文摘要 · Abstract (English)
Singing voice synthesis (SVS) and singing voice conversion (SVC) have achieved remarkable progress in generating natural-sounding human singing. However, existing systems are restricted to human timbres and have limited ability to synthesize voices outside the human range, which are increasingly demanded in creative applications such as video games, movies, and virtual characters. We introduce Non-Human Singing Generation (NHSG), covering non-human singing voice synthesis (NHSVS) and non-human singing voice conversion (NHSVC), as a novel machine learning task for generating musically coherent singing with non-human timbral characteristics. NHSG is particularly challenging due to the scarcity of non-human singing data, the lack of symbolic alignment, and the wide timbral gap between human and non-human voices. To address these challenges, we propose CartoonSing, a unified framework that integrates singing voice synthesis and conversion while bridging human and non-human singing generation. CartoonSing employs a two-stage pipeline: a score representation encoder trained with annotated human singing and a timbre-aware vocoder that reconstructs waveforms for both human and non-human audio. Experiments demonstrate that CartoonSing successfully generates non-human singing voices, generalizes to novel timbres, and extends conventional SVS and SVC toward creative, non-human singing generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。