用视觉线索提升电影音频分离效果,实现更精准的音轨拆分。
Cinematic Audio Source Separation Using Visual Cues
- 基于视觉信息的条件生成模型,融合音视频多模态数据。
- 在合成与真实电影数据上均表现优异,超越纯音频方法。
- 适合影视后期制作、配音及音频修复场景使用。
电影音频源分离(CASS)旨在将混合电影音频分解为语音、音乐和音效,支持配音和重制等应用。现有方法均为纯音频驱动,忽略了电影中声音与视觉高度一致的特点。本文提出首个音视频协同的电影音频分离框架(AV-CASS),利用视觉上下文提升分离质量。方法将CASS建模为条件流匹配的生成任务,实现多模态音频分离。针对缺乏带独立音轨的电影数据集的问题,设计了一种合成数据生成流程,将真实视频流(如人脸视频用于语音,场景视频用于音效)与音频配对,并构建专用视觉编码器处理双流输入。模型仅在合成数据上训练,却能有效泛化至真实电影内容,在合成、真实世界及纯音频基准上均取得优秀性能。代码与演示见:https://cass-flowmatching.github.io。
原文摘要 · Abstract (English)
Cinematic Audio Source Separation (CASS) aims to decompose mixed film audio into speech, music, and sound effects, enabling applications like dubbing and remastering. Existing CASS approaches are audio-only, overlooking the inherent audio-visual nature of films, where sounds often align with visual cues. We present the first framework for audio-visual CASS (AV-CASS), leveraging visual context to enhance separation quality. Our method formulates CASS as a conditional generative modeling problem using conditional flow matching, enabling multimodal audio source separation. To address the lack of cinematic datasets with isolated sound tracks, we introduce a training data synthesis pipeline that pairs in-the-wild audio and video streams (e.g., facial videos for speech, scene videos for effects) and design a dedicated visual encoder for this dual-stream setup. Trained entirely on synthetic data, our model generalizes effectively to real-world cinematic content and achieves strong performance on synthetic, real-world, and audio-only CASS benchmarks. Code and demo are available at \url{https://cass-flowmatching.github.io}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。