arXiv:2604.05731cs.CV2026-04

自动化生成精准时空对齐的电影音效,提升沉浸感。

FoleyDesigner: Immersive Stereo Foley Generation with Precise Spatio-Temporal Alignment for Film Clips

  • 基于多智能体架构分析视频,实现音效与画面精准同步。
  • 在8类常见音效上达到优于基线的时空对齐效果。
  • 支持专业音频混音流程,适合影视后期创作者使用。

音效设计在提升电影沉浸式听觉体验中起关键作用,但手动创建时空对齐的音频仍耗时费力。本文提出FoleyDesigner,一种受专业音效制作流程启发的新框架,集成影片分析、可控音效生成和专业音频混音功能。该框架采用多智能体架构进行精确的时空分析,通过在视频帧中提取的时空线索训练的潜在扩散模型实现时空对齐,并结合大语言模型驱动的混合机制模拟影视后期制作实践。为解决电影领域高质量立体音频数据集匮乏问题,我们构建了FilmStereo——首个含空间元数据、精确时间戳和语义标注的八类常见音效的专业立体音频数据集。系统支持交互式用户控制,且与专业工作流无缝集成,兼容符合ITU-R BS.775标准的5.1声道杜比全景声系统,提供广泛创作灵活性。大量实验表明,本方法在时空对齐方面显著优于现有基线,且完全兼容专业影视制作标准。

原文摘要 · Abstract (English)

Foley art plays a pivotal role in enhancing immersive auditory experiences in film, yet manual creation of spatio-temporally aligned audio remains labor-intensive. We propose FoleyDesigner, a novel framework inspired by professional Foley workflows, integrating film clip analysis, spatio-temporally controllable Foley generation, and professional audio mixing capabilities. FoleyDesigner employs a multi-agent architecture for precise spatio-temporal analysis. It achieves spatio-temporal alignment through latent diffusion models trained on spatio-temporal cues extracted from video frames, combined with large language model (LLM)-driven hybrid mechanisms that emulate post-production practices in film industry. To address the lack of high-quality stereo audio datasets in film, we introduce FilmStereo, the first professional stereo audio dataset containing spatial metadata, precise timestamps, and semantic annotations for eight common Foley categories. For applications, the framework supports interactive user control while maintaining seamless integration with professional pipelines, including 5.1-channel Dolby Atmos systems compliant with ITU-R BS.775 standards, thereby offering extensive creative flexibility. Extensive experiments demonstrate that our method achieves superior spatio-temporal alignment compared to existing baselines, with seamless compatibility with professional film production standards. The project page is available at https://gekiii996.github.io/FoleyDesigner/ .

音效生成时空对齐电影音频扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。