让音乐精准匹配视频节奏与情绪，实现视听同步的自动配乐新方法
MuVi: Video-to-Music Generation with Semantic Alignment and Rhythmic Synchronization
- 通过视觉适配器提取视频时序语义特征，指导音乐生成
- 对比预训练提升音乐与视频的节奏同步性，尤其在乐句周期上表现突出
- 支持风格控制，适合影视配乐、短视频创作等场景
生成与视频内容高度契合的音乐是一项挑战，需理解视觉语义并使音乐的旋律、节奏与动态与视觉叙事协调。本文提出MuVi框架，通过专用视觉适配器提取时空相关特征，生成既符合视频情绪主题又匹配节奏与节奏变化的音乐。引入基于音乐乐句周期性的对比音乐-视觉预训练方案以增强同步性。此外，基于流匹配的音乐生成器具备上下文学习能力，可灵活控制生成音乐的风格与类型。实验表明，MuVi在音频质量和时间同步性方面均优于现有方法。生成样例可在https://muvi-v2m.github.io 获取。
原文摘要 · Abstract (English)
Generating music that aligns with the visual content of a video has been a challenging task, as it requires a deep understanding of visual semantics and involves generating music whose melody, rhythm, and dynamics harmonize with the visual narratives. This paper presents MuVi, a novel framework that effectively addresses these challenges to enhance the cohesion and immersive experience of audio-visual content. MuVi analyzes video content through a specially designed visual adaptor to extract contextually and temporally relevant features. These features are used to generate music that not only matches the video's mood and theme but also its rhythm and pacing. We also introduce a contrastive music-visual pre-training scheme to ensure synchronization, based on the periodicity nature of music phrases. In addition, we demonstrate that our flow-matching-based music generator has in-context learning ability, allowing us to control the style and genre of the generated music. Experimental results show that MuVi demonstrates superior performance in both audio quality and temporal synchronization. The generated music video samples are available at https://muvi-v2m.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。