构建首个视频生成钢琴音乐的多模态评测基准，解决同步难题。
Towards Video to Piano Music Generation with Chain-of-Perform Support Benchmarks
- 通过分步演奏链（CoP）标注，实现视频与音频精准对齐
- 包含详细多模态标注，支持复杂交互评估
- 开源数据集与榜单，适合音乐生成与跨模态研究者
从视频生成高质量钢琴音乐需精确匹配视觉线索与音乐输出，确保语义与时间上的同步。然而现有评估数据集未能充分反映钢琴音乐生成所需的精细同步要求。为此，我们提出链式演奏（Chain-of-Perform, CoP）基准数据集——一个完全开源、专用于视频引导钢琴音乐生成的多模态基准。该数据集具备三大特性：(1) 详尽的多模态标注，通过分步演奏链指导实现视频内容与钢琴音频间的精确语义与时间对齐；(2) 兼容通用与专用任务的评估框架，支持严谨性能评测；(3) 完全开源数据集、标注与评估协议。数据集已公开于 https://github.com/acappemin/Video-to-Audio-and-Piano，配套持续更新的排行榜，推动该领域研究进展。
原文摘要 · Abstract (English)
Generating high-quality piano audio from video requires precise synchronization between visual cues and musical output, ensuring accurate semantic and temporal alignment.However, existing evaluation datasets do not fully capture the intricate synchronization required for piano music generation. A comprehensive benchmark is essential for two primary reasons: (1) existing metrics fail to reflect the complexity of video-to-piano music interactions, and (2) a dedicated benchmark dataset can provide valuable insights to accelerate progress in high-quality piano music generation. To address these challenges, we introduce the CoP Benchmark Dataset-a fully open-sourced, multimodal benchmark designed specifically for video-guided piano music generation. The proposed Chain-of-Perform (CoP) benchmark offers several compelling features: (1) detailed multimodal annotations, enabling precise semantic and temporal alignment between video content and piano audio via step-by-step Chain-of-Perform guidance; (2) a versatile evaluation framework for rigorous assessment of both general-purpose and specialized video-to-piano generation tasks; and (3) full open-sourcing of the dataset, annotations, and evaluation protocols. The dataset is publicly available at https://github.com/acappemin/Video-to-Audio-and-Piano, with a continuously updated leaderboard to promote ongoing research in this domain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。