提出新方法减少3D重建中的幻觉伪影,提升模型真实感。
HAD: Hallucination-Aware Diffusion Priors for 3D Reconstruction

- 用预训练的多视角合成网络评估生成图像的幻觉程度。
- 通过像素级幻觉评分屏蔽不可靠区域,避免伪影进入3D模型。
- 融合多视角条件生成结果,利用全局上下文提升重建质量。
扩散先验在稀疏视角3D重建中能通过生成新视角增强模型质量,但会引入与输入不一致的幻觉内容。为此,我们提出幻觉感知扩散先验(HAD),利用预先训练于大规模3D数据的前馈式新视角合成(NVS)网络,估计生成图像的像素级幻觉得分图。这些得分用于在渐进式3D重建过程中选择性地屏蔽不可靠像素,防止不存在的伪影进入最终模型。为进一步提升性能,我们在每个新视点上基于不同输入视图生成多个增强图像,并将其融合为综合图像,以利用所有输入视图的全局上下文信息。实验表明,该方法显著降低了扩散辅助3D重建中的幻觉伪影,在多个新视角合成基准上达到当前最优表现。项目代码与资源已公开于项目网站。
原文摘要 · Abstract (English)
Diffusion priors have recently demonstrated strong capability in enhancing the quality of sparse-view 3D reconstruction by augmenting training views at novel viewpoints, but they inevitably introduce hallucinated content -- artifacts inconsistent with the input views -- into the final 3D model. To address this challenge, we propose Hallucination-Aware Diffusion prior (HAD), which estimates pixel-wise hallucination score maps for augmented images by leveraging multi-view reasoning capabilities from a feedforward novel view synthesis (NVS) network pre-trained on large-scale 3D data. These hallucination scores enable selective masking of unreliable pixels during the progressive 3D reconstruction procedure, preventing the introduction of non-existent artifacts into the 3D model. To further enhance performance, we create multiple versions of augmented images at each novel view by conditioning the diffusion prior on different input views, which are then fused into a final image that leverages the broader context across all input views. We show that our method substantially reduces hallucination artifacts in diffusion-assisted 3D reconstruction, thereby achieving state-of-the-art performance across multiple benchmarks on novel view synthesis. Our project are publicly available at \href{https://xiliu8006.github.io/HAD-Project-website/}{project website}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。