用音频生成带时间戳的钢琴乐谱，更准更直观。
Rubato: Transcribing Piano Music with Timestamps
- 用提示条件编码器-解码器模型直接输出带时间戳的乐谱。
- 在钢琴乐谱转写任务上准确率超越现有最佳方法。
- 适合音乐学习、表演分析和音乐学研究者使用。
我们研究将音乐录音转换为带有时间戳的人类可读乐谱。此类输出能让听众清晰可视化演奏中的自由节奏（rubato），帮助学习者对比演奏与乐谱的时序差异，供音乐学者跨录音比较同一作品的演绎风格。本文提出(1)一种提示条件的编码器-解码器模型Rubato，用于输出(2)一种专为序列到序列训练设计的新式多声部音乐文本表示InterMo。实验表明，Rubato从音频生成带时间戳的钢琴乐谱，在记谱准确性上优于现有最优方法（基于级联架构）。即使给级联方法提供真实MIDI而非音频，Rubato仍表现更优，说明现有方法的上限主要源于表示方式而非声学感知。此外，由于Rubato在多个相关任务上联合训练（通过提示），其在简单任务如MIDI音符定位和节拍/强拍检测上也达到或超过最优单任务系统水平。演示地址：https://nctamer.github.io/rubato-transcription。
原文摘要 · Abstract (English)
We consider the conversion of musical recordings into human-readable sheet music annotated with timestamps. Such output lets a listener clearly visualize rubato (temporally expressive playing), a learner diagnose ensemble precision and timing choices against the written music, and a musicology scholar compare performance styles across recordings of the same work. We introduce (1) a prompt-conditioned encoder-decoder model, named Rubato, trained to output (2) a new textual representation for polyphonic music, named InterMo, which we designed for compatibility with sequence-to-sequence training. Our experiments demonstrate that Rubato produces timestamped piano sheet music from audio with higher notational accuracy than the best existing approaches, which are based on cascades. We find that even if the cascade is given ground-truth MIDI instead of audio, Rubato performs better, suggesting that the ceiling of existing approaches is primarily representational, not acoustic. Further, because Rubato is trained on several related tasks (with prompts), it competes with or outperforms the best single-task systems on related but simpler tasks like MIDI note grounding and beat/downbeat detection. A demo is available at https://nctamer.github.io/rubato-transcription .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。