用2D分割图微调预训练NeRF,实现高质量3D物体分割
InvNeRF-Seg: Fine-Tuning a Pre-Trained NeRF for 3D Object Segmentation
- 先训练标准NeRF,再用2D分割掩码微调,不改架构和损失函数
- 在合成水果和真实大豆数据集上优于SA3D和FruitNeRF
- 无需重新标注点云,计算开销小,适合需高精度3D分割的场景
神经辐射场(NeRF)已被广泛用于从2D RGB图像重建高质量3D点云。然而,对这些重建的3D场景进行分割对于物体计数、尺寸估计和场景理解等下游任务至关重要。直接在原始3D点云上使用深度学习进行分割需要耗时的人工标注;而直接在二值掩码上训练NeRF则因缺乏颜色和阴影线索导致几何学习失败。我们提出不变神经辐射场分割方法(InvNeRF-Seg),一种两步零修改微调策略。首先在RGB图像上训练标准NeRF,然后仅使用2D分割掩码进行微调,不改变模型架构或损失函数。该方法可从优化后的辐射场中直接生成更高质量、更干净的分割点云,且计算开销极低。密度场分析显示:物体区域密度上升,背景密度被抑制,确保分割结果清晰可解释。我们在合成水果和真实大豆数据集上验证了其优于SA3D和FruitNeRF的性能,有效将2D分割扩展至高质量3D分割。
原文摘要 · Abstract (English)
Neural Radiance Fields (NeRF) have been widely adopted for reconstructing high quality 3D point clouds from 2D RGB images. However, the segmentation of these reconstructed 3D scenes is more essential for downstream tasks such as object counting, size estimation, and scene understanding. While segmentation on raw 3D point clouds using deep learning requires labor intensive and time-consuming manual annotation, directly training NeRF on binary masks also fails due to the absence of color and shading cues essential for geometry learning. We propose Invariant NeRF for Segmentation (InvNeRFSeg), a two step, zero change fine tuning strategy for 3D segmentation. We first train a standard NeRF on RGB images and then fine tune it using 2D segmentation masks without altering either the model architecture or loss function. This approach produces higher quality, cleaner segmented point clouds directly from the refined radiance field with minimal computational overhead or complexity. Field density analysis reveals consistent semantic refinement: densities of object regions increase while background densities are suppressed, ensuring clean and interpretable segmentations. We demonstrate InvNeRFSegs superior performance over both SA3D and FruitNeRF on both synthetic fruit and real world soybean datasets. This approach effectively extends 2D segmentation to high quality 3D segmentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。