arXiv:2502.13128cs.SDcs.AI2025-02ICML被引 51

单阶段模型一键生成带歌词和伴奏的歌曲,支持精细控制音乐风格。

SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation

  • 统一自回归框架,直接生成人声与伴奏混合或分轨输出。
  • 支持歌词、乐器、风格、情绪等多维度控制,可选语音克隆。
  • 开源模型与数据处理流程,便于研究复现与社区协作。

文本到歌曲生成任务需从文本输入生成人声与伴奏,因领域复杂且数据稀缺而面临挑战。现有方法多采用多阶段流程,导致训练与推理繁琐,并因阶段间误差累积影响整体质量。本文提出 SongGen,一个全开源的单阶段自回归变压器模型,用于可控歌曲生成。该模型可对歌词、乐器描述、流派、情绪、音色等多元音乐属性进行细粒度控制,还支持可选的三秒参考音频实现语音克隆。在统一自回归框架下,SongGen 支持两种输出模式:混合模式直接生成人声与伴奏混合音轨,双轨模式分别生成人声与伴奏以增强下游应用灵活性。我们探索了不同令牌模式策略,带来显著性能提升并获得重要洞见。此外,设计了自动化数据预处理流程并配备有效质量控制机制。为促进社区参与与后续研究,将公开模型权重、训练代码、标注数据及预处理工具。代码已开源至 https://github.com/LiuZH-19/SongGen。

原文摘要 · Abstract (English)

Text-to-song generation, the task of creating vocals and accompaniment from textual inputs, poses significant challenges due to domain complexity and data scarcity. Existing approaches often employ multi-stage generation procedures, leading to cumbersome training and inference pipelines, as well as suboptimal overall generation quality due to error accumulation across stages. In this paper, we propose SongGen, a fully open-source, single-stage auto-regressive transformer designed for controllable song generation. The proposed model facilitates fine-grained control over diverse musical attributes, including lyrics and textual descriptions of instrumentation, genre, mood, and timbre, while also offering an optional three-second reference clip for voice cloning. Within a unified auto-regressive framework, SongGen supports two output modes: mixed mode, which generates a mixture of vocals and accompaniment directly, and dual-track mode, which synthesizes them separately for greater flexibility in downstream applications. We explore diverse token pattern strategies for each mode, leading to notable improvements and valuable insights. Furthermore, we design an automated data preprocessing pipeline with effective quality control. To foster community engagement and future research, we will release our model weights, training code, annotated data, and preprocessing pipeline. The code is available at https://github.com/LiuZH-19/SongGen.

文本生成音乐自回归模型语音克隆开源项目

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。