用程序化规则生成海量乐理推理题,提升AI识谱与创作能力。
Towards an AI Musician: Synthesizing Sheet Music Problems for Musical Reasoning
- 将乐理规则转为可执行函数,自动生成多样化乐谱推理题。
- 在SSMR-Bench上,强化推理后模型表现显著提升,跨基准泛化能力强。
- 适合研究音乐理解、多模态大模型及生成式创作的开发者和学者。
提升大语言模型(LLM)与多模态大语言模型(MLLM)对五线谱的理解能力,是构建AI音乐家的关键一步。然而,当前研究既缺乏乐理推理的评估基准,也缺少训练数据。受数学启发,我们提出将节拍、音程等核心乐理规则视为程序化函数,系统性地合成大量可验证的乐谱推理问题。该方法构建了数据生成框架,可生成文本与视觉双模态的乐谱问题,进而推出合成乐理推理基准(SSMR-Bench)及配套训练集。在SSMR-Bench上的评估表明,推理能力在识谱理解中至关重要,但模型对视觉谱面仍存挑战。通过强化学习与合成数据(RLVR)训练,所有模型在SSMR-Bench上均显著提升,且在已有的人工构造基准如MusicTheoryBench和MMMU音乐子集上也表现出明显进步。最终结果还显示,增强的推理能力可有效支持音乐创作。
原文摘要 · Abstract (English)
Enhancing the ability of Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) to interpret sheet music is a crucial step toward building AI musicians. However, current research lacks both evaluation benchmarks and training data for sheet music reasoning. Inspired by mathematics, where simple operations yield infinite verifiable problems, we introduce a novel approach that treats core music theory rules, such as those governing beats and intervals, as programmatic functions to systematically synthesize a vast and diverse corpus of sheet music reasoning problems. This approach allows us to introduce a data synthesis framework that generates verifiable sheet music questions in both textual and visual modalities, leading to the Synthetic Sheet Music Reasoning Benchmark (SSMR-Bench) and a complementary training set. Evaluation results on SSMR-Bench highlight the key role reasoning plays in interpreting sheet music, while also pointing out the ongoing challenges in understanding sheet music in a visual format. By leveraging synthetic data for RLVR, all models show significant improvements on the SSMR-Bench. Additionally, they also demonstrate considerable advancements on previously established human-crafted benchmarks, such as MusicTheoryBench and the music subset of MMMU. Finally, our results show that the enhanced reasoning ability can also facilitate music composition.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。