arXiv:2508.00782cs.GRcs.AI2025-08被引 3

利用声音的空间特性生成与音频空间一致的视频,提升视觉还原度。

SpA2V: Harnessing Spatial Auditory Cues for Audio-driven Spatially-aware Video Generation

论文配图:SpA2V: Harnessing Spatial Auditory Cues for Audio-driven Spatially-aware Video Generation
图 1 · 摘自论文原文
  • 从音频中提取音量、频率等空间线索,构建视频场景布局
  • 在无需训练的情况下,将布局引导注入扩散模型生成视频
  • 适合需要精确空间对齐的音频驱动视频应用

音频驱动视频生成旨在合成与输入音频同步的逼真视频,类似人类听声识景的能力。然而现有方法多聚焦于语义信息(如发声源类别),忽略声音固有的空间属性(如位置、运动方向),导致生成内容空间不准确。我们提出SpA2V,首个显式利用声音物理特性(如响度、频率)生成空间感知视频的框架。该框架分两阶段:1)音频引导视频规划:使用先进多模态大模型解析音频中的空间与语义线索,生成视频场景布局(VSL)作为中间表示;2)布局引导视频生成:设计高效方法将VSL作为条件引导注入预训练扩散模型,实现无训练的布局约束生成。大量实验表明,SpA2V能生成与输入音频在语义和空间上高度对齐的真实视频。

原文摘要 · Abstract (English)

Audio-driven video generation aims to synthesize realistic videos that align with input audio recordings, akin to the human ability to visualize scenes from auditory input. However, existing approaches predominantly focus on exploring semantic information, such as the classes of sounding sources present in the audio, limiting their ability to generate videos with accurate content and spatial composition. In contrast, we humans can not only naturally identify the semantic categories of sounding sources but also determine their deeply encoded spatial attributes, including locations and movement directions. This useful information can be elucidated by considering specific spatial indicators derived from the inherent physical properties of sound, such as loudness or frequency. As prior methods largely ignore this factor, we present SpA2V, the first framework explicitly exploits these spatial auditory cues from audios to generate videos with high semantic and spatial correspondence. SpA2V decomposes the generation process into two stages: 1) Audio-guided Video Planning: We meticulously adapt a state-of-the-art MLLM for a novel task of harnessing spatial and semantic cues from input audio to construct Video Scene Layouts (VSLs). This serves as an intermediate representation to bridge the gap between the audio and video modalities. 2) Layout-grounded Video Generation: We develop an efficient and effective approach to seamlessly integrate VSLs as conditional guidance into pre-trained diffusion models, enabling VSL-grounded video generation in a training-free manner. Extensive experiments demonstrate that SpA2V excels in generating realistic videos with semantic and spatial alignment to the input audios.

音频生成空间感知扩散模型多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。