用信息论提升少样本下的3D重建质量
MutualNeRF: Improve the Performance of NeRF under Limited Samples with Mutual Information Theory
- 以互信息统一衡量图像间关联，兼顾语义与像素级相关性
- 稀疏采样时最小化互信息，自动选择信息量大的新视角
- 少样本合成中最大化互信息，提升生成图像的准确性
本文提出MutualNeRF，一种基于互信息理论的NeRF增强框架，旨在改善有限样本下的3D场景重建性能。尽管NeRF在3D场景合成上表现优异，但在数据稀缺情况下仍面临挑战，现有方法引入先验知识却缺乏统一理论支持。我们引入互信息作为统一度量标准，从宏观（语义）和微观（像素）两个层面评估图像间相关性。针对稀疏视图采样，通过最小化互信息策略，在无需真实图像的前提下，自适应选择包含更多非重叠场景信息的新视角，采用贪心算法实现近似最优解。对于少样本视图合成，通过最大化推断图像与真实图像间的互信息，借助高效可插拔正则项，使推断图像能更充分地获取已知图像中的有效信息。在多种有限样本设置下的实验表明，该框架持续优于当前最优基线方法，验证了其有效性。
原文摘要 · Abstract (English)
This paper introduces MutualNeRF, a framework enhancing Neural Radiance Field (NeRF) performance under limited samples using Mutual Information Theory. While NeRF excels in 3D scene synthesis, challenges arise with limited data and existing methods that aim to introduce prior knowledge lack theoretical support in a unified framework. We introduce a simple but theoretically robust concept, Mutual Information, as a metric to uniformly measure the correlation between images, considering both macro (semantic) and micro (pixel) levels. For sparse view sampling, we strategically select additional viewpoints containing more non-overlapping scene information by minimizing mutual information without knowing ground truth images beforehand. Our framework employs a greedy algorithm, offering a near-optimal solution. For few-shot view synthesis, we maximize the mutual information between inferred images and ground truth, expecting inferred images to gain more relevant information from known images. This is achieved by incorporating efficient, plug-and-play regularization terms. Experiments under limited samples show consistent improvement over state-of-the-art baselines in different settings, affirming the efficacy of our framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。