让3D场景识别同时看懂已知和未知物体,突破传统标签限制。
AGO: Adaptive Grounding for Open World 3D Occupancy Prediction

- 用相似性训练实现3D伪标签自适应对齐,灵活应对开放世界。
- 在零样本和少样本下显著提升未知物体识别效果,闭世界性能领先4.09 mIoU。
- 适合做开放环境3D感知的科研与工程人员,尤其关注跨模态对齐。
开放世界3D语义占据预测旨在从传感器输入生成体素化3D表示,并识别已知与未知物体。将视觉语言模型(VLM)中的开放词汇知识迁移至该任务具有前景,但面临挑战。基于VLM生成2D伪标签的传统方法受限于预定义标签空间,缺乏泛化能力;而直接对齐预训练图像嵌入则因VLM中图像与文本表征不一致导致性能不可靠。为此,我们提出AGO框架,通过自适应定位机制处理多样化的开放世界场景。首先分别将周围图像和类别提示编码为3D与文本嵌入,利用3D伪标签进行基于相似性的接地训练;此外,引入模态适配器将3D嵌入映射到与VLM图像嵌入对齐的空间,缓解模态差异。在Occ3D-nuScenes数据集上的实验表明,AGO在零样本与少样本迁移下均显著提升未知物体预测性能,且在闭世界自监督设置中达到当前最优表现,优于先前方法4.09 mIoU。代码已开源:https://github.com/EdwardLeeLPZ/AGO。
原文摘要 · Abstract (English)
Open-world 3D semantic occupancy prediction aims to generate a voxelized 3D representation from sensor inputs while recognizing both known and unknown objects. Transferring open-vocabulary knowledge from vision-language models (VLMs) offers a promising direction but remains challenging. However, methods based on VLM-derived 2D pseudo-labels with traditional supervision are limited by a predefined label space and lack general prediction capabilities. Direct alignment with pretrained image embeddings, on the other hand, often fails to achieve reliable performance because of inconsistent image and text representations in VLMs. To address these challenges, we propose AGO, a novel 3D occupancy prediction framework with adaptive grounding to handle diverse open-world scenarios. AGO first encodes surrounding images and class prompts into 3D and text embeddings, respectively, leveraging similarity-based grounding training with 3D pseudo-labels. Additionally, a modality adapter maps 3D embeddings into a space aligned with VLM-derived image embeddings, reducing modality gaps. Experiments on Occ3D-nuScenes show that AGO improves unknown object prediction in zero-shot and few-shot transfer while achieving state-of-the-art closed-world self-supervised performance, surpassing prior methods by 4.09 mIoU. Code is available at: https://github.com/EdwardLeeLPZ/AGO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。