让音频生成精准跟随视频中特定物体，提升音效制作可控性。
Video Object Segmentation-Aware Audio Generation
- 用物体分割图+视频+文本联合控制音频生成
- 在音乐演奏视频上实现高保真、定位精准的音效合成
- 适合专业音效师和需要精细控制的多媒体创作
现有跨模态音频生成模型缺乏精确用户控制，限制了其在专业音效制作流程中的应用。这些模型通常关注整个视频画面，无法有效聚焦特定物体，导致生成冗余背景音或错误对象的声音。为此，我们提出视频物体分割感知的音频生成新任务，显式利用物体级分割图作为声音合成的条件。我们提出SAGANet，一种新型多模态生成模型，通过结合视觉分割掩码、视频帧与文本提示，实现对音频生成的细粒度、视觉定位控制。为支持该任务及后续研究，我们构建了包含分割信息的音乐乐器演奏视频基准数据集Segmented Music Solos。实验表明，该方法显著优于当前最先进水平，为可控、高保真音效合成树立了新标准。代码、样例与数据集已公开。
原文摘要 · Abstract (English)
Existing multimodal audio generation models often lack precise user control, which limits their applicability in professional Foley workflows. In particular, these models focus on the entire video and do not provide precise methods for prioritizing a specific object within a scene, generating unnecessary background sounds, or focusing on the wrong objects. To address this gap, we introduce the novel task of video object segmentation-aware audio generation, which explicitly conditions sound synthesis on object-level segmentation maps. We present SAGANet, a new multimodal generative model that enables controllable audio generation by leveraging visual segmentation masks along with video and textual cues. Our model provides users with fine-grained and visually localized control over audio generation. To support this task and further research on segmentation-aware Foley, we propose Segmented Music Solos, a benchmark dataset of musical instrument performance videos with segmentation information. Our method demonstrates substantial improvements over current state-of-the-art methods and sets a new standard for controllable, high-fidelity Foley synthesis. Code, samples, and Segmented Music Solos are available at https://saganet.notion.site
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。