arXiv:2412.11428cs.CV2024-12AAAI被引 2

用稳定扩散模型选关键视角,提升3D重建对视角变换的鲁棒性。

View Transformation Robustness for Multi-View 3D Object Reconstruction with Reconstruction Error-Guided View Selection

  • 基于重建误差引导选择最能覆盖缺陷的视角
  • 在大幅视角变换下仍保持高重建精度
  • 适合需要抗视角变化的3D重建应用

视图变换鲁棒性(VTR)对基于深度学习的多视角3D物体重建至关重要,体现模型在不同视角变换输入下的稳定性。现有研究较少关注此问题。提升VTR的一种直接方法是增加带多种视角变换的数据进行训练。近期大视觉模型(如Stable Diffusion)可仅凭单图生成3D模型或合成新视角图像,但直接部署于推理阶段计算开销大,且其视角变换鲁棒性未被保证。为充分利用Stable Diffusion能力而不增加推理负担,本文提出用其生成新视角以增强VTR。不同于随机生成视角,我们提出一种重建误差引导的视角选择方法,根据3D预测的重建误差空间分布,选择能最大程度覆盖误差区域的视角。在包含大幅视角变换的数据集上训练和测试,验证了模型的鲁棒性。大量实验表明,该方法优于当前最优3D重建方法及其他对比方法。代码已开源:https://github.com/zqyq/VTR。

原文摘要 · Abstract (English)

View transformation robustness (VTR) is critical for deep-learning-based multi-view 3D object reconstruction models, which indicates the methods' stability under inputs with various view transformations. However, existing research seldom focused on view transformation robustness in multi-view 3D object reconstruction. One direct way to improve the models' VTR is to produce data with more view transformations and add them to model training. Recent progress on large vision models, particularly Stable Diffusion models, has provided great potential for generating 3D models or synthesizing novel view images with only a single image input. Directly deploying these models at inference consumes heavy computation resources and their robustness to view transformations is not guaranteed either. To fully utilize the power of Stable Diffusion models without extra inference computation burdens, we propose to generate novel views with Stable Diffusion models for better view transformation robustness. Instead of synthesizing random views, we propose a reconstruction error-guided view selection method, which considers the reconstruction errors' spatial distribution of the 3D predictions and chooses the views that could cover the reconstruction errors as much as possible. The methods are trained and tested on sets with large view transformations to validate the 3D reconstruction models' robustness to view transformations. Extensive experiments demonstrate that the proposed method can outperform state-of-the-art 3D reconstruction methods and other view transformation robustness comparison methods. Code is available at: https://github.com/zqyq/VTR.

3D重建视角鲁棒性扩散模型视角选择

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。