arXiv:2412.10768cs.CVcs.MM2024-12CVPR被引 21

联合视频与文本生成完整音效,包含屏幕内外声音。

VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation

  • 采用流模型与双模态编码器,协同视频和文本生成音效。
  • 在636对数据上验证,联合模态显著提升音效完整性。
  • 适合影视音效合成、多模态内容生成研究者使用。

近期音频生成研究聚焦于文本到音频(T2A)和视频到音频(V2A)任务,但二者均无法生成包含屏幕内外声音的完整音效:T2A难以对齐画面内物体声音,V2A则缺失语义完整的离屏音效。本文提出全场景音频生成任务——给定视频与文本提示,生成与视频时间同步、语义上与文本和视频一致的屏幕内外声音。为克服以往联合图文生成方法的模态偏差问题,我们提出基于流的Transformer模型VinTAGe,包含视觉-文本编码器与联合VT-SiT模型,并引入预训练的单模态生成模型进行额外引导以减少偏差。由于缺乏合适基准,我们构建了包含636个视频-文本-音频对的VinTAGe-Bench数据集,其中涵盖屏幕内外声音。在该数据集上的实验表明,联合图文交互对实现完整音效生成至关重要。此外,VinTAGe在VGGSound基准上达到当前最优性能。代码与预训练模型将公开,演示视频见:https://www.youtube.com/watch?v=QmqWhUjPkJI。

原文摘要 · Abstract (English)

Recent advances in audio generation have focused on text-to-audio (T2A) and video-to-audio (V2A) tasks. However, T2A or V2A methods cannot generate holistic sounds (onscreen and off-screen). This is because T2A cannot generate sounds aligning with onscreen objects, while V2A cannot generate semantically complete (offscreen sounds missing). In this work, we address the task of holistic audio generation: given a video and a text prompt, we aim to generate both onscreen and offscreen sounds that are temporally synchronized with the video and semantically aligned with text and video. Previous approaches for joint text and video-to-audio generation often suffer from modality bias, favoring one modality over the other. To overcome this limitation, we introduce VinTAGe, a flow-based transformer model that jointly considers text and video to guide audio generation. Our framework comprises two key components: a Visual-Text Encoder and a Joint VT-SiT model. To reduce modality bias and improve generation quality, we employ pretrained uni-modal text-to-audio and video-to-audio generation models for additional guidance. Due to the lack of appropriate benchmarks, we also introduce VinTAGe-Bench, a dataset of 636 video-text-audio pairs containing both onscreen and offscreen sounds. Our comprehensive experiments on VinTAGe-Bench demonstrate that joint text and visual interaction is necessary for holistic audio generation. Furthermore, VinTAGe achieves state-of-the-art results on the VGGSound benchmark. Our source code and pre-trained models will be released. Demo is available at: https://www.youtube.com/watch?v=QmqWhUjPkJI.

音频生成多模态视频理解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。