通过自监督与标签监督结合,实现可控的视频音乐检索。
Semi-Supervised Contrastive Learning for Controllable Video-to-Music Retrieval
- 融合自监督与标签监督对比学习,构建音视频联合嵌入空间。
- 利用音乐流派标签提升检索效果,支持多维度控制检索权重。
- 适用于影视配乐、短视频背景音乐等场景,可灵活调节检索侧重。
内容创作者常需为视频匹配合适的音乐,如电影配乐、视频博客及社交媒体内容的背景音乐,但这一过程耗时费力。为此,我们提出一种新框架,可自动为给定视频检索匹配的音乐片段,反之亦然。该方法利用标注的音乐标签及视觉与音乐元素间的艺术对应关系。不同于以往跨模态音乐检索工作,本方法结合自监督与监督训练目标,采用自监督和标签监督的对比学习来训练音视频联合嵌入空间。实验表明,使用音乐流派标签作为监督信号有效提升了性能,且该框架可推广至其他音乐标注(如情感、乐器等)。此外,方法在推理阶段支持细粒度控制自监督与标签信息的权重比例。我们在多种视频到音乐及音乐到视频检索任务上评估了所学嵌入表示,结果验证了该方法在可控音乐-视频检索中的有效性。
原文摘要 · Abstract (English)
Content creators often use music to enhance their videos, from soundtracks in movies to background music in video blogs and social media content. However, identifying the best music for a video can be a difficult and time-consuming task. To address this challenge, we propose a novel framework for automatically retrieving a matching music clip for a given video, and vice versa. Our approach leverages annotated music labels, as well as the inherent artistic correspondence between visual and music elements. Distinct from previous cross-modal music retrieval works, our method combines both self-supervised and supervised training objectives. We use self-supervised and label-supervised contrastive learning to train a joint embedding space between music and video. We show the effectiveness of our approach by using music genre labels for the supervised training component, and our framework can be generalized to other music annotations (e.g., emotion, instrument, etc.). Furthermore, our method enables fine-grained control over how much the retrieval process focuses on self-supervised vs. label information at inference time. We evaluate the learned embeddings through a variety of video-to-music and music-to-video retrieval tasks. Our experiments show that the proposed approach successfully combines self-supervised and supervised objectives and is effective for controllable music-video retrieval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。