arXiv:2511.15308cs.CV2025-11TPAMI

用自然语言精确定位3D点云,支持复杂描述和城市级场景。

Text2Loc++: Generalizing 3D Point Cloud Localization from Natural Language

  • 分步定位:先粗后细,融合语言与点云的多层级语义理解。
  • 在KITTI360Pose上提升15%,对复杂描述和多样环境均表现稳健。
  • 新数据集+新方法,适合跨模态定位、自动驾驶等应用研究者。

本文解决使用复杂多样的自然语言描述定位3D点云子图的问题,提出Text2Loc++模型,采用从粗到细的定位流程实现语言与点云的有效跨模态对齐。为支持基准测试,构建了一个覆盖多种城市场景的城市场景级数据集,包含彩色与非彩色点云,并将位置描述划分为三个语言复杂度层级。全局定位阶段,结合预训练语言模型与带最大池化的层次化变换器(HTM)捕捉句级语义,通过基于注意力的点云编码器实现空间理解;提出遮蔽实例训练(MIT)过滤未对齐对象,提升多模态鲁棒性;引入模态感知的层次对比学习(MHCL),融合跨模态、子图、文本及实例级损失以优化嵌入空间。精细定位阶段,摒弃显式文本-实例匹配,设计轻量但强大的原型式地图克隆(PMC)与级联交叉注意力变换器(CCAT)框架。在KITTI360Pose数据集上的大量实验表明,Text2Loc++相比现有方法性能最高提升15%。此外,模型在新数据集上表现出强泛化能力,能有效处理复杂语言表达与多样化城市环境。代码与数据集将公开共享。

原文摘要 · Abstract (English)

We tackle the problem of localizing 3D point cloud submaps using complex and diverse natural language descriptions, and present Text2Loc++, a novel neural network designed for effective cross-modal alignment between language and point clouds in a coarse-to-fine localization pipeline. To support benchmarking, we introduce a new city-scale dataset covering both color and non-color point clouds from diverse urban scenes, and organize location descriptions into three levels of linguistic complexity. In the global place recognition stage, Text2Loc++ combines a pretrained language model with a Hierarchical Transformer with Max pooling (HTM) for sentence-level semantics, and employs an attention-based point cloud encoder for spatial understanding. We further propose Masked Instance Training (MIT) to filter out non-aligned objects and improve multimodal robustness. To enhance the embedding space, we introduce Modality-aware Hierarchical Contrastive Learning (MHCL), incorporating cross-modal, submap-, text-, and instance-level losses. In the fine localization stage, we completely remove explicit text-instance matching and design a lightweight yet powerful framework based on Prototype-based Map Cloning (PMC) and a Cascaded Cross-Attention Transformer (CCAT). Extensive experiments on the KITTI360Pose dataset show that Text2Loc++ outperforms existing methods by up to 15%. In addition, the proposed model exhibits robust generalization when evaluated on the new dataset, effectively handling complex linguistic expressions and a wide variety of urban environments. The code and dataset will be made publicly available.

3D定位跨模态点云自然语言

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。