用生成模型模拟拍摄不一致，提升3D重建鲁棒性
SimVS: Simulating World Inconsistencies for Robust View Synthesis
- 用生成视频模型模拟真实拍摄中的光照、运动等不一致
- 在真实场景不一致下实现高精度静态3D重建
- 适合处理非专业拍摄的复杂真实场景
新视角合成技术在静态场景中表现优异，但在非专业拍摄条件下（如光照变化、场景运动等）因难以建模而表现下降。本文提出利用生成视频模型模拟拍摄过程中的世界不一致性，结合现有多视角数据集，生成用于训练多视角统一网络的合成数据。该网络能将不一致观测融合为一致的3D场景。实验表明，相比传统增强方法，本方法在应对真实场景变化时显著更优，可实现多种挑战性不一致条件下的高精度静态3D重建。
原文摘要 · Abstract (English)
Novel-view synthesis techniques achieve impressive results for static scenes but struggle when faced with the inconsistencies inherent to casual capture settings: varying illumination, scene motion, and other unintended effects that are difficult to model explicitly. We present an approach for leveraging generative video models to simulate the inconsistencies in the world that can occur during capture. We use this process, along with existing multi-view datasets, to create synthetic data for training a multi-view harmonization network that is able to reconcile inconsistent observations into a consistent 3D scene. We demonstrate that our world-simulation strategy significantly outperforms traditional augmentation methods in handling real-world scene variations, thereby enabling highly accurate static 3D reconstructions in the presence of a variety of challenging inconsistencies. Project page: https://alextrevithick.github.io/simvs
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。