用音频直接生成音乐描述,比传统方法更灵活高效。
Rethinking Music Captioning with Music Metadata LLMs
- 先从音频预测详细元数据,再用预训练大模型转成自然语言描述
- 训练时间更短,性能与端到端模型相当
- 可自由调整描述风格,还能补全缺失的音乐元数据
音乐描述任务旨在生成对音乐内容的自然语言描述,有助于音乐理解与可控音乐生成。但高质量音乐描述数据稀缺,通常依赖大语言模型(LLMs)从元数据(如流派、情绪等)合成训练数据,存在风格固化和事实与风格混淆的问题。为此,本文提出基于元数据的音乐描述新方法:先训练一个元数据预测模型从音频中推断详细元数据,再在推理时通过预训练的LLM将其转换为生动的描述。相比使用LLM生成元数据标签并训练端到端模型的基线方法,本方法:(1)训练时间更短,性能相当;(2)训练后可灵活调整输出风格,满足不同质量与风格需求;(3)支持输入音频和部分元数据进行元数据补全或填充,适用于音乐数据整理场景。
原文摘要 · Abstract (English)
Music captioning, or the task of generating a natural language description of music, is useful for both music understanding and controllable music generation. Training captioning models, however, typically requires high-quality music caption data which is scarce compared to metadata (e.g., genre, mood, etc.). As a result, it is common to use large language models (LLMs) to synthesize captions from metadata to generate training data for captioning models, though this process imposes a fixed stylization and entangles factual information with natural language style. As a more direct approach, we propose metadata-based captioning. We train a metadata prediction model to infer detailed music metadata from audio and then convert it into expressive captions via pre-trained LLMs at inference time. Compared to a strong end-to-end baseline trained on LLM-generated captions derived from metadata, our method: (1) achieves comparable performance in less training time over end-to-end captioners, (2) offers flexibility to easily change stylization post-training, enabling output captions to be tailored to specific stylistic and quality requirements, and (3) can be prompted with audio and partial metadata to enable powerful metadata imputation or in-filling--a common task for organizing music data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。