arXiv:2510.22141cs.CVcs.CL2025-10被引 5

用语言引导实现3D占位预测,能识别未知物体且无需额外训练数据。

LOC: A General Language-Guided Framework for Open-Set 3D Occupancy Prediction

  • 通过语言提示融合多帧点云,用KNN赋予体素语义。
  • 在nuScenes上对已知类精度高,未知类也能准确区分。
  • 支持自监督与有监督学习,适合开放集场景理解任务。

视觉-语言模型在开放集挑战中表现优异,但3D数据集有限制约其在3D场景理解中的应用。我们提出LOC,一种通用的语言引导框架,可适配多种占位网络,支持监督与自监督学习。针对自监督任务,采用多帧激光雷达点云融合策略,利用泊松重建填补空洞,并通过K近邻(KNN)为体素分配语义,获得完整的体素表示。为缓解高维特征直接蒸馏导致的特征同质化问题,引入密集对比学习(DCL),利用密集体素语义信息和预设文本提示,高效提升开放集识别能力,无需像素级精细标注。本框架还可利用已有真实标签进一步优化性能。模型将密集体素特征嵌入CLIP特征空间,结合文本与图像像素信息进行分类,基于文本与语义相似性判别。在nuScenes数据集上的实验表明,该方法在已知类别上实现高精度预测,并能在不增加训练数据的情况下有效区分未知类别。

原文摘要 · Abstract (English)

Vision-Language Models (VLMs) have shown significant progress in open-set challenges. However, the limited availability of 3D datasets hinders their effective application in 3D scene understanding. We propose LOC, a general language-guided framework adaptable to various occupancy networks, supporting both supervised and self-supervised learning paradigms. For self-supervised tasks, we employ a strategy that fuses multi-frame LiDAR points for dynamic/static scenes, using Poisson reconstruction to fill voids, and assigning semantics to voxels via K-Nearest Neighbor (KNN) to obtain comprehensive voxel representations. To mitigate feature over-homogenization caused by direct high-dimensional feature distillation, we introduce Densely Contrastive Learning (DCL). DCL leverages dense voxel semantic information and predefined textual prompts. This efficiently enhances open-set recognition without dense pixel-level supervision, and our framework can also leverage existing ground truth to further improve performance. Our model predicts dense voxel features embedded in the CLIP feature space, integrating textual and image pixel information, and classifies based on text and semantic similarity. Experiments on the nuScenes dataset demonstrate the method's superior performance, achieving high-precision predictions for known classes and distinguishing unknown classes without additional training data.

3D占位语言引导开放集自监督

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。