用神经辐射场修复被遮挡图像,提升真实场景重建质量
NeRF-MIR: Towards High-Quality Restoration of Masked Images with Neural Radiance Fields
- 基于像素熵设计射线发射策略,更好融合多视角信息
- 自迭代恢复机制让遮挡区域逐步重建,效果优于现有方法
- 专为遮挡图像设计,适合做3D场景修复的研究者使用
神经辐射场(NeRF)在新视角合成中表现优异,但在基于受损图像重建三维场景方面仍有提升空间,而自然场景拍摄中图像损坏很常见,严重影响NeRF性能。本文提出NeRF-MIR,一种专为遮挡图像修复设计的新型神经渲染方法,展现了NeRF在该领域的潜力。针对传统随机射线发射难以学习复杂纹理的问题,提出基于补丁熵的射线发射策略(PERE),有效分配射线以融合多视角信息。此外,引入渐进式迭代恢复机制(PIRE),通过自训练过程逐步修复遮挡区域。同时设计动态加权损失函数,自动调整遮挡区域的损失权重。由于现有数据集不支持基于NeRF的遮挡图像修复,本文构建了三个模拟损坏场景的遮挡数据集。在真实数据和构造数据上的大量实验表明,NeRF-MIR在遮挡图像修复方面显著优于现有方法。
原文摘要 · Abstract (English)
Neural Radiance Fields (NeRF) have demonstrated remarkable performance in novel view synthesis. However, there is much improvement room on restoring 3D scenes based on NeRF from corrupted images, which are common in natural scene captures and can significantly impact the effectiveness of NeRF. This paper introduces NeRF-MIR, a novel neural rendering approach specifically proposed for the restoration of masked images, demonstrating the potential of NeRF in this domain. Recognizing that randomly emitting rays to pixels in NeRF may not effectively learn intricate image textures, we propose a \textbf{P}atch-based \textbf{E}ntropy for \textbf{R}ay \textbf{E}mitting (\textbf{PERE}) strategy to distribute emitted rays properly. This enables NeRF-MIR to fuse comprehensive information from images of different views. Additionally, we introduce a \textbf{P}rogressively \textbf{I}terative \textbf{RE}storation (\textbf{PIRE}) mechanism to restore the masked regions in a self-training process. Furthermore, we design a dynamically-weighted loss function that automatically recalibrates the loss weights for masked regions. As existing datasets do not support NeRF-based masked image restoration, we construct three masked datasets to simulate corrupted scenarios. Extensive experiments on real data and constructed datasets demonstrate the superiority of NeRF-MIR over its counterparts in masked image restoration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。