简单整合音视频扩散模型，实现高质量同步生成
A Simple but Strong Baseline for Sounding Video Generation: Effective Adaptation of Audio and Video Diffusion Models for Joint Generation
- 通过时间步调整与跨模态位置编码，实现音视频生成对齐
- 在多个数据集上优于现有方法，音视频同步性显著提升
- 适合想快速搭建音视频联合生成系统的研究人员
本文构建了一个简单但强大的音视频生成基线。基于已有的音频和视频扩散模型，我们引入额外模块将其整合为单一模型，并训练其联合生成音视频。为增强音视频对的对齐效果，提出两种新机制：一是时间步调整，为各基础模型提供不同的时间步信息，以对齐跨模态生成过程；二是跨模态条件作为位置编码（CMC-PE），将跨模态信息嵌入为时间位置信号，类似位置编码输入模型。相比常用的交叉注意力机制，CMC-PE为生成数据提供了更优的时间对齐归纳偏置。实验验证了两种机制的有效性，且本方法在多个基准上优于现有方法。
原文摘要 · Abstract (English)
In this work, we build a simple but strong baseline for sounding video generation. Given base diffusion models for audio and video, we integrate them with additional modules into a single model and train it to make the model jointly generate audio and video. To enhance alignment between audio-video pairs, we introduce two novel mechanisms in our model. The first one is timestep adjustment, which provides different timestep information to each base model. It is designed to align how samples are generated along with timesteps across modalities. The second one is a new design of the additional modules, termed Cross-Modal Conditioning as Positional Encoding (CMC-PE). In CMC-PE, cross-modal information is embedded as if it represents temporal position information, and the embeddings are fed into the model like positional encoding. Compared with the popular cross-attention mechanism, CMC-PE provides a better inductive bias for temporal alignment in the generated data. Experimental results validate the effectiveness of the two newly introduced mechanisms and also demonstrate that our method outperforms existing methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。