让自动驾驶感知系统识别未见过的物体,且保持高精度与实时性。
Open-Vocabulary BEV Segmentation with 3D-Aware Geometric Constraints

- 用视觉语言模型结合3D几何约束,实现开放词汇的鸟瞰图分割。
- 在nuScenes上对未知类别提升15.3 mIoU,性能超越闭集方法。
- 推理速度比传统方法快2.5倍,内存占用仅为0.22倍,适合部署。
鸟瞰图(BEV)感知将多摄像头图像融合为统一的俯视表示,用于自动驾驶。尽管近期进展显著,现有方法仍局限于封闭集场景,难以应对真实世界中的不可预测环境。本文提出开放词汇鸟瞰图分割(OVBS),利用视觉语言模型(VLMs)识别训练集外类别,同时保持精确的BEV感知和实时效率。关键挑战在于2D VLM语义向BEV投影时固有的3D几何不一致性。为此,我们提出OVBEVSeg框架,通过三阶段渐进式设计增强基于高斯点阵的高效反投影:(1) 借助可靠3D投影进行2D到BEV伪标签生成,支持开放词汇泛化;(2) 联合2D-BEV场景级优化并引入BEV结构约束,确保3D几何一致性;(3) 通过3D几何蒸馏实现在线高效推理。在nuScenes数据集上,OVBEVSeg达到当前最优性能,在未见类别上较闭集方法提升15.3 mIoU。即使无新类别标注,其表现仍可媲美使用高达40%标注数据的自/半监督基线。此外,推理速度提升2.5倍,内存消耗仅为投影类方法的0.22倍。
原文摘要 · Abstract (English)
Bird's-eye view (BEV) perception fuses multi-camera images into a unified top-down representation for autonomous driving. Despite recent progress, state-of-the-art methods remain confined to closed-set scenarios, making them vulnerable to unpredictable real-world environments. In this work, we introduce open-vocabulary BEV segmentation (OVBS), which leverages vision-language models (VLMs) to recognize categories beyond the training set while maintaining precise BEV perception and real-time efficiency. A key challenge in OVBS lies in the 3D geometric inconsistency inherent in the ill-posed lifting of 2D VLM semantics into BEV. To address this, we propose OVBEVSeg, a geometry-aware OVBS framework that enhances efficient Gaussian splatting (GS)-based unprojection by leveraging robust 3D geometric constraints across three progressive stages: (1) 2D-to-BEV pseudo-labeling via reliable 3D projection for OV generalization; (2) joint 2D-BEV per-scene optimization with BEV structural constraints for 3D geometric consistency; and (3) 3D geometric distillation for online efficiency. On the nuScenes dataset, OVBEVSeg achieves state-of-the-art performance, outperforming closed-set methods by 15.3 mIoU on unseen categories. Remarkably, even with no novel-class ground-truth labels, it remains competitive with self- and semi-supervised baselines trained with up to 40% of ground-truth annotations. Furthermore, it achieves 2.5x faster inference with only 0.22x the memory consumption of projection-based methods. Project page: https://hchoi256.github.io/projects/ovbevseg/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。