首个端到端视频扩散模型,分离静态外观与动态运动。
DiViD: Disentangled Video Diffusion for Static-Dynamic Factorization
- 用全局静态令牌和帧级动态令牌显式分解视频内容。
- 在真实数据集上交换准确率最高,交叉泄漏减少37%以上。
- 适合需要精细运动控制的视频生成与编辑任务。
无监督地分离视频中的静态外观与动态运动仍是基础挑战,现有基于变分自编码器(VAE)和生成对抗网络(GAN)的方法常因信息泄露和模糊重建而受限。我们提出DiViD,首个实现显式静态-动态解耦的端到端视频扩散框架。其序列编码器从首帧提取全局静态令牌,并生成每帧的动态令牌,显式移除运动码中的静态内容。条件去噪扩散概率模型(DDPM)解码器引入三项归纳偏置:共享噪声调度保证时序一致性;随时间变化的基于KL的瓶颈在早期紧缩(压缩静态信息),后期放松(丰富动态);交叉注意力将全局静态令牌传递至所有帧,同时保持动态令牌帧内独立。正交性正则化进一步防止残留的静态-动态泄露。我们在真实世界基准上使用交换准确性与交叉泄露指标评估DiViD,结果表明其优于最先进序列解耦方法:达到最高的交换联合准确率,保持静态保真度的同时提升动态迁移能力,平均交叉泄露降低37%以上。
原文摘要 · Abstract (English)
Unsupervised disentanglement of static appearance and dynamic motion in video remains a fundamental challenge, often hindered by information leakage and blurry reconstructions in existing VAE- and GAN-based approaches. We introduce DiViD, the first end-to-end video diffusion framework for explicit static-dynamic factorization. DiViD's sequence encoder extracts a global static token from the first frame and per-frame dynamic tokens, explicitly removing static content from the motion code. Its conditional DDPM decoder incorporates three key inductive biases: a shared-noise schedule for temporal consistency, a time-varying KL-based bottleneck that tightens at early timesteps (compressing static information) and relaxes later (enriching dynamics), and cross-attention that routes the global static token to all frames while keeping dynamic tokens frame-specific. An orthogonality regularizer further prevents residual static-dynamic leakage. We evaluate DiViD on real-world benchmarks using swap-based accuracy and cross-leakage metrics. DiViD outperforms state-of-the-art sequential disentanglement methods: it achieves the highest swap-based joint accuracy, preserves static fidelity while improving dynamic transfer, and reduces average cross-leakage.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。