构建钢琴音乐多模态联合嵌入模型，精准捕捉细微语义差异。
PianoBind: A Multimodal Joint Embedding Model for Pop-piano Music
- 设计多源训练策略，在音符、音频、文本三模态上联合建模。
- 在小规模同质钢琴数据集上实现优于通用模型的图文检索性能。
- 方法可复用于其他同质领域多模态学习，如乐器独奏分析。
独奏钢琴音乐虽为单一乐器形式，却能表达丰富的语义信息，涵盖风格、情绪与流派。然而，当前主流音乐表示模型多基于大规模数据训练，难以捕捉同质独奏钢琴音乐中的细微语义差异。此外，现有钢琴专用模型多为单模态，无法体现钢琴音乐在音频、乐谱符号与文本描述等多模态上的内在特性。为此，我们提出PianoBind——一种专用于钢琴的多模态联合嵌入模型。系统研究了在联合嵌入框架中针对小规模、同质钢琴数据集的多源训练策略与模态利用方式。实验表明，PianoBind学习到的多模态表示能有效捕捉钢琴音乐的细微特征，在域内与跨域钢琴数据集上均显著优于通用音乐联合嵌入模型的文本到音乐检索性能。同时，其设计思路为处理同质数据集的多模态表示学习提供了可复用的实践经验。
原文摘要 · Abstract (English)
Solo piano music, despite being a single-instrument medium, possesses significant expressive capabilities, conveying rich semantic information across genres, moods, and styles. However, current general-purpose music representation models, predominantly trained on large-scale datasets, often struggle to captures subtle semantic distinctions within homogeneous solo piano music. Furthermore, existing piano-specific representation models are typically unimodal, failing to capture the inherently multimodal nature of piano music, expressed through audio, symbolic, and textual modalities. To address these limitations, we propose PianoBind, a piano-specific multimodal joint embedding model. We systematically investigate strategies for multi-source training and modality utilization within a joint embedding framework optimized for capturing fine-grained semantic distinctions in (1) small-scale and (2) homogeneous piano datasets. Our experimental results demonstrate that PianoBind learns multimodal representations that effectively capture subtle nuances of piano music, achieving superior text-to-music retrieval performance on in-domain and out-of-domain piano datasets compared to general-purpose music joint embedding models. Moreover, our design choices offer reusable insights for multimodal representation learning with homogeneous datasets beyond piano music.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。