让神经辐射场持续学习多个场景,内存占用极低且不遗忘旧场景。
SCARF: Scalable Continual Learning Framework for Memory-efficient Multiple Neural Radiance Fields
- 用共享权重矩阵+场景专属权重,高效表示多个3D场景。
- 在三个数据集上实现当前最佳渲染质量,存储开销极小。
- 适合需要长期积累3D场景的工业应用,如数字孪生。
本文提出一种新型持续学习框架,用于合成多个场景的新视角图像,可增量式学习多个3D场景,并仅使用新场景的训练数据更新网络参数。基于神经辐射场(NeRF),其通过多层感知机建模场景的密度与辐射场。尽管NeRF及其扩展在单个场景中能生成逼真新视角,但高效管理不断增长的3D NeRF资产已成为新科学问题。极少工作关注多场景的高效表示或持续学习能力,而这对于NeRF的实际应用至关重要。核心思想是将多个场景表示为跨场景权重矩阵与由全局参数生成器产生的场景专属权重矩阵的线性组合。此外,提出不确定表面知识蒸馏策略,将旧场景的辐射场知识迁移到新模型。该权重矩阵表示显著降低内存需求,而不确定表面蒸馏有效缓解灾难性遗忘,保持旧场景的逼真渲染质量。实验表明,该方法在NeRF-Synthetic、LLFF和TanksAndTemples数据集上均达到当前最优持续学习渲染效果,同时存储成本极低。
原文摘要 · Abstract (English)
This paper introduces a novel continual learning framework for synthesising novel views of multiple scenes, learning multiple 3D scenes incrementally, and updating the network parameters only with the training data of the upcoming new scene. We build on Neural Radiance Fields (NeRF), which uses multi-layer perceptron to model the density and radiance field of a scene as the implicit function. While NeRF and its extensions have shown a powerful capability of rendering photo-realistic novel views in a single 3D scene, managing these growing 3D NeRF assets efficiently is a new scientific problem. Very few works focus on the efficient representation or continuous learning capability of multiple scenes, which is crucial for the practical applications of NeRF. To achieve these goals, our key idea is to represent multiple scenes as the linear combination of a cross-scene weight matrix and a set of scene-specific weight matrices generated from a global parameter generator. Furthermore, we propose an uncertain surface knowledge distillation strategy to transfer the radiance field knowledge of previous scenes to the new model. Representing multiple 3D scenes with such weight matrices significantly reduces memory requirements. At the same time, the uncertain surface distillation strategy greatly overcomes the catastrophic forgetting problem and maintains the photo-realistic rendering quality of previous scenes. Experiments show that the proposed approach achieves state-of-the-art rendering quality of continual learning NeRF on NeRF-Synthetic, LLFF, and TanksAndTemples datasets while preserving extra low storage cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。