构建跨符号体系的音乐多模态推理框架,验证不同记谱法间的结构一致性。
ONOTE: Hypergraph-Grounded Omnimodal Reasoning for Computational Music Science

- 基于超图构建音乐理论知识库,支持实体与关系检索
- 在四种任务中验证记谱、音频、符号输出的结构合规性
- 揭示感知与音乐理论应用之间的系统性差异
多模态乐谱处理以五线谱为中心,需在听觉、视觉、符号和物理表征间保持一致。现有工作在识别与转录任务上分散,极少测试不同记谱系统间的结构一致性。西方五线谱偏见和模型评估标准不明确掩盖了音高、节奏、顺序及乐器约束中的错误。我们提出ONOTE,一个统一框架,将音乐视为可度量的跨表征对应关系科学领域。其仅测试基准涵盖五线谱、简谱与吉他谱,覆盖多种风格、乐器与结构条件,并提供对齐的多模态衍生数据。四个互补任务包括乐谱理解、记谱转换、音频转录与符号生成,检验音高与时值顺序、输出语法及披露的乐器特定约束。ONOTE还从外部音乐理论材料构建带有溯源信息的命题超图,支持基于实体与超边的证据检索。确定性有效性检查、公开的结构合规性SMG评分及受控的RAG对比揭示了视觉识别与结构保持输出间的差距。结果区分了感知、音乐理论应用以及结构或物理约束满足程度。ONOTE为研究表示不变性与知识引导干预提供了可审计框架。
原文摘要 · Abstract (English)
Omnimodal notation processing, centered on sheet music, is a controlled scientific setting in which auditory, visual, symbolic, and physical representations must encode the same musical events. Yet existing work remains fragmented across recognition and transcription, rarely testing structural consistency across notation systems. Western-staff bias and underspecified model judges further conceal errors in pitch, timing, ordering, and instrument-specific constraints. We introduce ONOTE, a unified framework that treats music as a scientifically structured domain of measurable cross-representation correspondences. Its test-only benchmark draws on a diverse collection of musical sources covering staff, Jianpu, and tablature across varied genres, instruments, and structural conditions, with aligned multimodal derivatives. Four complementary tasks cover score understanding, notation conversion, audio transcription, and symbolic generation, testing pitch and duration ordering, output syntax, and disclosed instrument-specific constraints. ONOTE also constructs a provenance-bearing proposition hypergraph from external music-theory materials for entity- and hyperedge-based evidence retrieval. Deterministic validity checks, disclosed structural-compliance SMG scoring, and controlled RAG comparisons reveal gaps between visual recognition and structure-preserving outputs. Results separate perception from music-theory application and structural or physical constraint satisfaction. ONOTE provides an auditable framework for studying representation invariance and knowledge-grounded intervention in computational music science.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。