VersBand实现歌词、旋律、人声与伴奏的提示控制生成，支持多任务且对齐精准。
Versatile Framework for Song Generation with Prompt-based Control
- 分模块设计：人声、伴奏、歌词与旋律分别建模，支持多提示控制
- 人声生成快且质量高，伴奏通过专家选择机制提升对齐与控制性
- 适用于音乐创作、跨模态生成等场景，适合需要精细控制的创作者
歌曲生成致力于基于多种提示生成可控的高质量歌曲。然而，现有方法在人声与伴奏的提示控制及正确对齐方面表现不佳，且难以支持多样化任务。为此，我们提出 VersBand，一个用于合成高质量、对齐准确的提示控制歌曲的多任务生成框架。该框架包含三个核心模型：1）VocalBand，一种解耦模型，采用流匹配方法生成演唱风格、音高和梅尔频谱，实现快速、高质量的人声生成并支持风格控制；2）AccompBand，一种基于流的Transformer模型，引入 Band-MOE 机制，动态选择合适专家以提升质量、对齐性和控制性，实现与人声对齐的可控伴奏生成；3）两个生成模型：LyricBand 用于歌词生成，MelodyBand 用于旋律生成，共同构成支持多提示控制的完整系统。实验结果表明，VersBand 在多项歌曲生成任务中优于基线模型，涵盖客观与主观评估指标。演示与代码已公开于 https://aaronz345.github.io/VersBandDemo 与 https://github.com/AaronZ345/VersBand。
原文摘要 · Abstract (English)
Song generation focuses on producing controllable high-quality songs based on various prompts. However, existing methods struggle to generate vocals and accompaniments with prompt-based control and proper alignment. Additionally, they fall short in supporting various tasks. To address these challenges, we introduce VersBand, a multi-task song generation framework for synthesizing high-quality, aligned songs with prompt-based control. VersBand comprises these primary models: 1) VocalBand, a decoupled model, leverages the flow-matching method for generating singing styles, pitches, and mel-spectrograms, allowing fast, high-quality vocal generation with style control. 2) AccompBand, a flow-based transformer model, incorporates the Band-MOE, selecting suitable experts for enhanced quality, alignment, and control. This model allows for generating controllable, high-quality accompaniments aligned with vocals. 3) Two generation models, LyricBand for lyrics and MelodyBand for melodies, contribute to the comprehensive multi-task song generation system, allowing for extensive control based on multiple prompts. Experimental results show that VersBand outperforms baseline models across multiple song generation tasks using objective and subjective metrics. Demos and codes are available at https://aaronz345.github.io/VersBandDemo and https://github.com/AaronZ345/VersBand.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。