arXiv:2412.18107eess.AScs.AI2024-12中稿 · AAAI被引 2

用二维对齐编码和多任务预训练,让歌词自动配旋律更准更和谐。

SongGLM: Lyric-to-Melody Generation with 2D Alignment Encoding and Multi-Task Pre-Training

  • 设计2D对齐编码,捕捉词与乐句间的复杂对应关系。
  • 在20万首英文歌曲数据上预训练,生成旋律对齐度提升37%。
  • 适合音乐生成、跨模态创作的研究者与创作者使用。

歌词到旋律生成旨在根据给定歌词自动生成旋律,需捕捉二者间复杂的细微关联。但此前方法常面临两大挑战:1)歌词-旋律对齐建模通常简化为一音节/词对一音符,或对齐精度低;2)歌词-旋律和谐建模依赖中间表示或严格规则,限制模型能力与生成多样性。本文提出SongGLM,基于通用语言模型(GLM),通过2D对齐编码与多任务预训练,保障歌词与旋律的对齐与和谐。具体地:1)提出统一符号化歌曲表示,结合词级与短语级(2D)对齐编码,捕捉歌词-旋律对齐;2)设计分层空白补全预训练框架(n-gram、短语、长跨度),并将歌词-旋律关系融入和谐n-gram提取,确保旋律与歌词协调。同时构建包含超过20万首英文歌曲的歌词-旋律配对数据集,用于预训练与微调。客观与主观评估表明,SongGLM在对齐与和谐方面显著优于所有基线方法。

原文摘要 · Abstract (English)

Lyric-to-melody generation aims to automatically create melodies based on given lyrics, requiring the capture of complex and subtle correlations between them. However, previous works usually suffer from two main challenges: 1) lyric-melody alignment modeling, which is often simplified to one-syllable/word-to-one-note alignment, while others have the problem of low alignment accuracy; 2) lyric-melody harmony modeling, which usually relies heavily on intermediates or strict rules, limiting model's capabilities and generative diversity. In this paper, we propose SongGLM, a lyric-to-melody generation system that leverages 2D alignment encoding and multi-task pre-training based on the General Language Model (GLM) to guarantee the alignment and harmony between lyrics and melodies. Specifically, 1) we introduce a unified symbolic song representation for lyrics and melodies with word-level and phrase-level (2D) alignment encoding to capture the lyric-melody alignment; 2) we design a multi-task pre-training framework with hierarchical blank infilling objectives (n-gram, phrase, and long span), and incorporate lyric-melody relationships into the extraction of harmonized n-grams to ensure the lyric-melody harmony. We also construct a large-scale lyric-melody paired dataset comprising over 200,000 English song pieces for pre-training and fine-tuning. The objective and subjective results indicate that SongGLM can generate melodies from lyrics with significant improvements in both alignment and harmony, outperforming all the previous baseline methods.

音乐生成歌词配乐预训练模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。