用相机图像自监督训练激光雷达语义分割,减少人工标注依赖。
Multi-modal NeRF Self-Supervision for LiDAR Semantic Segmentation
- 通过神经辐射场从相机视角生成伪标签,实现跨模态自监督。
- 在nuScenes、SemanticKITTI等数据集上提升分割精度,优于纯监督方法。
- 适合缺乏标注数据的自动驾驶感知场景,尤其适用于多传感器融合。
激光雷达语义分割是自动驾驶感知的基础任务,需为每个点分配语义标签。现有全监督模型依赖大量人工标注,成本高昂且难以泛化。相机图像与激光雷达常共存,可借助通用2D基础模型(如SAM)提取信息,但因传感器位置和时间差异导致的视差效应带来域适应挑战。本文提出半监督学习框架,利用未标注激光雷达点云与来自相机图像的知识进行自监督。通过添加辅助神经辐射场(NeRF)头,从相机视角向体素特征投射光线,预测密度与语义逻辑值并渲染像素语义;同时使用Segment-Anything模型生成通用掩码,与渲染结果融合生成伪标签以监督模型。推理时移除NeRF头,仅用激光雷达输入。在nuScenes、SemanticKITTI和ScribbleKITTI三个公开数据集上验证了方法的有效性,显著提升性能。
原文摘要 · Abstract (English)
LiDAR Semantic Segmentation is a fundamental task in autonomous driving perception consisting of associating each LiDAR point to a semantic label. Fully-supervised models have widely tackled this task, but they require labels for each scan, which either limits their domain or requires impractical amounts of expensive annotations. Camera images, which are generally recorded alongside LiDAR pointclouds, can be processed by the widely available 2D foundation models, which are generic and dataset-agnostic. However, distilling knowledge from 2D data to improve LiDAR perception raises domain adaptation challenges. For example, the classical perspective projection suffers from the parallax effect produced by the position shift between both sensors at their respective capture times. We propose a Semi-Supervised Learning setup to leverage unlabeled LiDAR pointclouds alongside distilled knowledge from the camera images. To self-supervise our model on the unlabeled scans, we add an auxiliary NeRF head and cast rays from the camera viewpoint over the unlabeled voxel features. The NeRF head predicts densities and semantic logits at each sampled ray location which are used for rendering pixel semantics. Concurrently, we query the Segment-Anything (SAM) foundation model with the camera image to generate a set of unlabeled generic masks. We fuse the masks with the rendered pixel semantics from LiDAR to produce pseudo-labels that supervise the pixel predictions. During inference, we drop the NeRF head and run our model with only LiDAR. We show the effectiveness of our approach in three public LiDAR Semantic Segmentation benchmarks: nuScenes, SemanticKITTI and ScribbleKITTI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。