构建了含多模态数据的钢琴演奏数据集,支持动作与音乐协同研究。
SKY-Piano: A Multimodal Piano Performance Dataset

- 采集7位专业与12位业余演奏者11小时多模态数据,含动作、视频、音频、MIDI等
- 提供带标记和补全两种形式的动作数据,支持人体运动分析与重建
- 适用于音乐信息检索、演奏生成、指法预测等研究,适合多模态学习方向
钢琴演奏研究日益依赖音频与MIDI之外的多模态数据。我们提出SKY-Piano,一个包含7位专业与12位业余钢琴家共11小时演奏数据的多模态数据集,涵盖动作、多视角视频、音频、MIDI及MusicXML乐谱。曲目选择兼顾演奏技巧、难度与演奏者水平,覆盖共同核心曲库。动作数据包括手部与身体运动,提供带标记(因遮挡不可靠样本已标注)与插补(缺失数据已重建)两种形式,并同步提供Visual3D人体分段运动学数据及其他时序对齐模态。为便于快速浏览,我们开发了交互式网页浏览器。此外,我们构建了基于MIDI与动作数据的指法标注模型与工具,实现伪指法标注。最后,以微调实验展示将MIDI生成动作的应用案例。
原文摘要 · Abstract (English)
Music information retrieval research on piano performance increasingly involves diverse modalities of data and annotations beyond audio and MIDI. We present SKY-Piano, a multimodal piano performance dataset that includes 11 hours of performance recordings of motion, multi-view video, audio, MIDI from 7 professional and 12 amateur pianists along with MusicXML scores. The performance pieces were selected considering playing technique, difficulty, and performer expertise on a shared core repertoire. The motion data include both hand and body motion, released in both flagged form, where samples lost to marker occlusion are marked as unreliable, and imputed form, where those gaps are reconstructed, together with Visual3D body-segment kinematics and other time-synchronized modalities. To easily browse different modalities of data at a glance, we provide an interactive web browser. In addition, we developed a fingering annotation model and tool for deriving pseudo fingering annotations from the MIDI and motion data. Lastly, we present MIDI-to-motion generation through a fine-tuning experiment as a use case of the dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。