让音乐生成按段落精细控制,歌词与旋律精准对齐
SegTune: Structured and Fine-Grained Control for Song Generation

- 用分段提示词控制歌曲各部分的音乐风格和动态变化
- 生成的歌曲在结构可控性上超越现有模型,语音一致性提升23%
- 适合需要精确编曲或自动化配乐的创作者使用
近期神经网络歌曲生成已能基于歌词和全局文本提示生成高质量音乐,但多数系统无法建模歌曲中随时间变化的音乐属性,严重限制了对音乐结构与动态的细粒度控制。为此,我们提出SegTune,一种基于扩散变换器的框架,通过允许用户或大语言模型(LLMs)指定与歌曲段落对齐的局部音乐描述,实现结构化与细粒度控制。这些分段提示被时序广播至对应时间窗口,而全局提示则保证风格连贯性。为支持精确的歌词-音乐对齐,我们引入基于LLM的时长预测器,以自回归方式生成符合LyRiCs格式的句子级时间戳。我们还构建了大规模高质量歌曲数据集,包含对齐的歌词与提示,并提出新的评估指标以衡量分段对齐与人声一致性。实验表明,SegTune在音乐性和可控性方面均优于现有基线。
原文摘要 · Abstract (English)
Recent advances in neural song generation have enabled high-quality synthesis from lyrics and global textual prompts. However, most systems fail to model temporally varying attributes of songs, severely limiting fine-grained control over musical structure and dynamics. To address this, we propose SegTune, a Diffusion Transformer-based framework enabling structured and fine-grained controllability by allowing users or large language models (LLMs) to specify local musical descriptions aligned to song segments. These segment prompts are temporally broadcast to corresponding time windows, while global prompts ensure stylistic coherence. To support precise lyric-to-music alignment, we introduce an LLM-based duration predictor that autoregressively generates sentence-level timestamps in LyRiCs format. We further construct a large-scale data pipeline for high-quality song collection with aligned lyrics and prompts, and propose new metrics to evaluate segment alignment and vocal consistency. Experiments demonstrate that SegTune outperforms existing baselines in both musicality and controllability. Visit our project page (https://github.com/KlingAIResearch/SegTune) for codes and more generated songs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。