用自监督学习实现3D场景中物体的精准定位，适合机器人和AR设备部署。
Locate 3D: Real-World Object Localization via Self-Supervised Learning in 3D
- 基于3D-JEPA自监督算法，从RGB-D流中学习点云上下文特征。
- 在13万+标注数据上训练，3D定位准确率刷新基准线。
- 直接处理传感器原始数据，适用于真实世界机器人与AR应用。
我们提出LOCATE 3D，一种从指代表达（如“沙发与灯之间的小型咖啡桌”）中定位3D场景物体的模型。该模型在标准指代定位基准上达到新最优性能，并展现出强大的泛化能力。关键在于3D-JEPA——一种适用于传感器点云的新型自监督学习算法。它利用2D基础模型（CLIP、DINO）对3D点云进行特征提取，并以潜在空间中的掩码预测作为预训练任务，学习上下文感知的点云特征。训练完成后，3D-JEPA编码器与语言条件解码器联合微调，共同预测3D掩码和边界框。此外，我们构建了包含超过13万标注的LOCATE 3D DATASET，覆盖多种采集设置，支持系统性泛化研究并强化模型性能。
原文摘要 · Abstract (English)
We present LOCATE 3D, a model for localizing objects in 3D scenes from referring expressions like "the small coffee table between the sofa and the lamp." LOCATE 3D sets a new state-of-the-art on standard referential grounding benchmarks and showcases robust generalization capabilities. Notably, LOCATE 3D operates directly on sensor observation streams (posed RGB-D frames), enabling real-world deployment on robots and AR devices. Key to our approach is 3D-JEPA, a novel self-supervised learning (SSL) algorithm applicable to sensor point clouds. It takes as input a 3D pointcloud featurized using 2D foundation models (CLIP, DINO). Subsequently, masked prediction in latent space is employed as a pretext task to aid the self-supervised learning of contextualized pointcloud features. Once trained, the 3D-JEPA encoder is finetuned alongside a language-conditioned decoder to jointly predict 3D masks and bounding boxes. Additionally, we introduce LOCATE 3D DATASET, a new dataset for 3D referential grounding, spanning multiple capture setups with over 130K annotations. This enables a systematic study of generalization capabilities as well as a stronger model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。