arXiv:2411.17489cs.CVcs.AI2024-11ICCV被引 4

提出一种新度量方法,可无参考地定位3D重建中的图像瑕疵。

Puzzle Similarity: A Perceptually-guided Cross-Reference Metric for Artifact Detection in 3D Scene Reconstructions

  • 基于训练视图的图像块统计建立场景特异性分布,识别新视角中重建不佳区域。
  • 在无人工标注参考的情况下,对瑕疵定位效果超越现有方法,与人类判断高度一致。
  • 适用于稀疏输入重建、自动修复和引导采集等任务,推动高质量3D重建落地。

现代重建技术能从稀疏2D视图有效建模复杂3D场景,但缺乏真实图像作为基准,且无参考度量难以可靠预测伪影图,导致新视角质量评估困难,限制了修补等后处理技术的应用。为解决此问题,近期研究引入交叉参考类度量,仅利用多视角上下文预测图像质量(arXiv:2404.14409)。本文提出新度量方法 Puzzle Similarity,通过训练视图的图像块统计建立场景特定分布,用于识别新视图中的劣质重建区域。由于缺乏评估交叉参考方法的基准,我们构建了一个新的、由人工标注的未见重建视图中伪影与失真图数据集。实验表明,该方法在无对齐参考条件下,实现新视角伪影定位的最先进性能,且与人类评估高度相关。该度量可应用于自动图像修复、引导采集或稀疏输入3D重建等场景。

原文摘要 · Abstract (English)

Modern reconstruction techniques can effectively model complex 3D scenes from sparse 2D views. However, automatically assessing the quality of novel views and identifying artifacts is challenging due to the lack of ground truth images and the limitations of no-reference image metrics in predicting reliable artifact maps. The absence of such metrics hinders assessment of the quality of novel views and limits the adoption of post-processing techniques, such as inpainting, to enhance reconstruction quality. To tackle this, recent work has established a new category of metrics (cross-reference), predicting image quality solely by leveraging context from alternate viewpoint captures (arXiv:2404.14409). In this work, we propose a new cross-reference metric, Puzzle Similarity, which is designed to localize artifacts in novel views. Our approach utilizes image patch statistics from the training views to establish a scene-specific distribution, later used to identify poorly reconstructed regions in the novel views. Given the lack of good measures to evaluate cross-reference methods in the context of 3D reconstruction, we collected a novel human-labeled dataset of artifact and distortion maps in unseen reconstructed views. Through this dataset, we demonstrate that our method achieves state-of-the-art localization of artifacts in novel views, correlating with human assessment, even without aligned references. We can leverage our new metric to enhance applications like automatic image restoration, guided acquisition, or 3D reconstruction from sparse inputs. Find the project page at https://nihermann.github.io/puzzlesim/ .

3D重建伪影检测无参考度量交叉参考

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。