XMusic用多模态提示生成情感可控的高质量音乐
XMusic: Towards a Generalized and Controllable Symbolic Music Generation Framework
- 将图像、文字等提示转化为音乐情绪、节奏等符号元素
- 通过多任务评估筛选出高质量音乐，效果超越现有方法
- 适合音乐创作、AI艺术生成及跨模态内容设计者
近年来，人工智能生成内容（AIGC）在图像和文本生成领域取得显著进展，生成内容已接近人类水平。然而，AI生成音乐的质量仍未达到同等标准，主要受限于情感控制难与输出质量低。本文提出通用符号音乐生成框架XMusic，支持图像、视频、文本、标签和哼唱等多种灵活提示，生成情感可控且高质量的符号化音乐。XMusic包含两个核心组件：XProjector将多模态提示映射到投影空间中的音乐元素（情绪、流派、节奏、音符），实现精准匹配；XComposer由生成器与选择器构成，生成器基于创新的符号音乐表示生成情感可控且悦耳的音乐，选择器通过质量评估、情绪识别与流派识别的多任务学习机制筛选优质作品。此外，我们构建了大规模标注数据集XMIDI，含108,023个带精确情绪与流派标签的MIDI文件。客观与主观评估均表明，XMusic显著优于当前最优方法，音乐质量突出。该项目获2023世界人工智能大会（WAIC）九大收藏亮点之一。项目主页：https://xmusic-project.github.io。
原文摘要 · Abstract (English)
In recent years, remarkable advancements in artificial intelligence-generated content (AIGC) have been achieved in the fields of image synthesis and text generation, generating content comparable to that produced by humans. However, the quality of AI-generated music has not yet reached this standard, primarily due to the challenge of effectively controlling musical emotions and ensuring high-quality outputs. This paper presents a generalized symbolic music generation framework, XMusic, which supports flexible prompts (i.e., images, videos, texts, tags, and humming) to generate emotionally controllable and high-quality symbolic music. XMusic consists of two core components, XProjector and XComposer. XProjector parses the prompts of various modalities into symbolic music elements (i.e., emotions, genres, rhythms and notes) within the projection space to generate matching music. XComposer contains a Generator and a Selector. The Generator generates emotionally controllable and melodious music based on our innovative symbolic music representation, whereas the Selector identifies high-quality symbolic music by constructing a multi-task learning scheme involving quality assessment, emotion recognition, and genre recognition tasks. In addition, we build XMIDI, a large-scale symbolic music dataset that contains 108,023 MIDI files annotated with precise emotion and genre labels. Objective and subjective evaluations show that XMusic significantly outperforms the current state-of-the-art methods with impressive music quality. Our XMusic has been awarded as one of the nine Highlights of Collectibles at WAIC 2023. The project homepage of XMusic is https://xmusic-project.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。