用文本生成图像解决激光雷达开放词汇语义分割难题
IGLOSS: Image Generation for Lidar Open-vocabulary Semantic Segmentation
- 通过文本生成图像构建类别原型，避免图像-文本模态鸿沟
- 在nuScenes和SemanticKITTI上达到当前最优性能
- 适合做3D感知中未见类识别的开发者与研究者
本文提出一种零样本开放词汇语义分割（OVSS）新方法，用于3D车载激光雷达数据。为克服基于视觉语言模型（如CLIP）存在的图像-文本模态差距，该方法转而利用文本生成图像来创建类别原型。基于从2D视觉基础模型（VFM）蒸馏得到的3D网络，通过将点云中3D点特征与这些原型图像的2D特征进行匹配实现标签预测。该方法在nuScenes和SemanticKITTI数据集上均达到当前最优水平。代码、预训练模型及生成图像已开源：https://github.com/valeoai/IGLOSS。
原文摘要 · Abstract (English)
This paper presents a new method for the zero-shot open-vocabulary semantic segmentation (OVSS) of 3D automotive lidar data. To circumvent the recognized image-text modality gap that is intrinsic to approaches based on Vision Language Models (VLMs) such as CLIP, our method relies instead on image generation from text, to create prototype images. Given a 3D network distilled from a 2D Vision Foundation Model (VFM), we then label a point cloud by matching 3D point features with 2D image features of these prototypes. Our method is state-of-the-art for OVSS on nuScenes and SemanticKITTI. Code, pre-trained models, and generated images are available at https://github.com/valeoai/IGLOSS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。