用双流结构和符号提示，提升音乐练习错音检测准确率
LadderSym: A Multimodal Interleaved Transformer for Music Practice Error Detection
- 双流编码器加跨模态对齐，强化音频与乐谱对比能力
- 在MAESTRO-E数据集上，漏奏音检测F1提升至56.3%（翻倍）
- 适合音乐教育、智能伴奏系统开发者参考
音乐学习者可通过精准的错音检测工具获益。现有方法通常通过启发式或可学习模型将音频录音与乐谱对比。本文提出LadderSym，一种基于Transformer的新型音乐错音检测方法。该方法基于两个关键观察：(1) 晚期融合限制了跨流对齐与多模态比较能力；(2) 依赖乐谱-音频匹配会在同时出现多个音符时引入频谱模糊，降低性能。为解决这些问题，LadderSym引入：(1) 双流编码器配合跨流对齐模块，增强音频比对能力与错误检测F1分数；(2) 多模态策略，将符号乐谱作为解码器提示，减少歧义并提升精度。我们在MAESTRO-E和CocoChorales-E数据集上评估，按音符类别测量F1分数。相比之前最优方法，LadderSym在MAESTRO-E上漏奏音检测F1从26.8%提升至56.3%，多奏音检测提升14.4个百分点（72.0%→86.4%）。类似提升也在CocoChorales-E上观测到。此外，我们在自建真实数据上也进行了评估。本工作为序列评估任务提供了关于对比模型的新见解，适用于强化学习、人类技能评估与模型评价。代码见：https://github.com/ben2002chou/LadderSYM
原文摘要 · Abstract (English)
Music learners can greatly benefit from tools that accurately detect errors in their practice. Existing approaches typically compare audio recordings to music scores using heuristics or learnable models. This paper introduces LadderSym, a novel Transformer-based method for music error detection. LadderSym is guided by two key observations about the state-of-the-art approaches: (1) late fusion limits inter-stream alignment and cross-modality comparison capability; and (2) reliance on score audio introduces ambiguity in the frequency spectrum, degrading performance in music with concurrent notes. To address these limitations, LadderSym introduces (1) a two-stream encoder with inter-stream alignment modules to improve audio comparison capabilities and error detection F1 scores, and (2) a multimodal strategy that leverages both audio and symbolic scores by incorporating symbolic representations as decoder prompts, reducing ambiguity and improving F1 scores. We evaluate our method on the MAESTRO-E and CocoChorales-E datasets by measuring the F1 score for each note category. Compared to the previous state of the art, LadderSym more than doubles F1 for missed notes on MAESTRO-E (26.8% -> 56.3%) and improves extra note detection by 14.4 points (72.0% -> 86.4%). Similar gains are observed on CocoChorales-E. Furthermore, we also evaluate our models on real data we curated. This work introduces insights about comparison models that could inform sequence evaluation tasks for reinforcement learning, human skill assessment, and model evaluation. Code: https://github.com/ben2002chou/LadderSYM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。