提升多视角视频生成的时空一致性，实现更逼真的4D动态资产输出。
SV4D 2.0: Enhancing Spatio-Temporal Consistency in Multi-View Video Diffusion for High-Quality 4D Generation
- 重构网络结构，消除参考视图依赖，优化3D与帧注意力融合机制。
- 在新视角视频合成中细节损失降低14%，4D一致性指标下降44%。
- 适合需要高质量动态3D内容生成的研究者与工业应用开发者。
我们提出Stable Video 4D 2.0（SV4D 2.0），一种用于动态3D资产生成的多视角视频扩散模型。相比前代SV4D，SV4D 2.0对遮挡和大运动更具鲁棒性，泛化能力更强，且在细节清晰度与时空一致性方面生成质量更高。通过四大改进：1）网络架构：消除参考多视角依赖，设计3D与帧注意力融合机制；2）数据：提升训练数据的质量与数量；3）训练策略：采用渐进式3D-4D训练以增强泛化；4）4D优化：通过两阶段精修与渐进帧采样处理3D不一致与大运动问题。大量实验表明，SV4D 2.0在视觉与量化指标上均有显著提升，在新视角视频合成中，细节损失降低14\\%（LPIPS），4D一致性下降44\\%（FV4D）；在4D优化任务中，分别降低12\\%（LPIPS）与24\\%（FV4D）。项目页面：https://sv4d20.github.io。
原文摘要 · Abstract (English)
We present Stable Video 4D 2.0 (SV4D 2.0), a multi-view video diffusion model for dynamic 3D asset generation. Compared to its predecessor SV4D, SV4D 2.0 is more robust to occlusions and large motion, generalizes better to real-world videos, and produces higher-quality outputs in terms of detail sharpness and spatio-temporal consistency. We achieve this by introducing key improvements in multiple aspects: 1) network architecture: eliminating the dependency of reference multi-views and designing blending mechanism for 3D and frame attention, 2) data: enhancing quality and quantity of training data, 3) training strategy: adopting progressive 3D-4D training for better generalization, and 4) 4D optimization: handling 3D inconsistency and large motion via 2-stage refinement and progressive frame sampling. Extensive experiments demonstrate significant performance gain by SV4D 2.0 both visually and quantitatively, achieving better detail (-14\% LPIPS) and 4D consistency (-44\% FV4D) in novel-view video synthesis and 4D optimization (-12\% LPIPS and -24\% FV4D) compared to SV4D. Project page: https://sv4d20.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。