用大模型+空间推理实现零样本3D物体定位，无需训练数据
SORT3D: Spatial Object-centric Reasoning Toolbox for Zero-Shot 3D Grounding Using Large Language Models
- 结合2D属性与启发式空间推理，利用大模型进行逐步推理解析
- 在两个基准上达到顶尖零样本性能，支持视点依赖的复杂定位
- 可实时运行于自动驾驶车辆，适用于未见过的真实场景导航
理解带空间关系和属性的物体指代语言并将其在3D中定位，对人机协作机器人至关重要。但该任务因场景多样、物体细粒度多、语言描述自由而困难。此外，3D领域难以获取大量自然语言训练数据。因此，方法需少样本学习并零样本泛化至新环境。我们提出SORT3D，利用2D数据中的丰富物体属性，融合启发式空间推理工具箱与大语言模型（LLM）的序列推理能力。重要的是，该方法无需文本到3D数据训练，可零样本应用于未见环境。实验表明，SORT3D在两个基准上的复杂视点依赖定位任务中达到当前最优零样本性能。我们还实现了系统在两辆自动驾驶汽车上的实时运行，并验证其可在未见真实环境中实现物体目标导航。所有源码已公开于https://github.com/nzantout/SORT3D。
原文摘要 · Abstract (English)
Interpreting object-referential language and grounding objects in 3D with spatial relations and attributes is essential for robots operating alongside humans. However, this task is often challenging due to the diversity of scenes, large number of fine-grained objects, and complex free-form nature of language references. Furthermore, in the 3D domain, obtaining large amounts of natural language training data is difficult. Thus, it is important for methods to learn from little data and zero-shot generalize to new environments. To address these challenges, we propose SORT3D, an approach that utilizes rich object attributes from 2D data and merges a heuristics-based spatial reasoning toolbox with the ability of large language models (LLMs) to perform sequential reasoning. Importantly, our method does not require text-to-3D data for training and can be applied zero-shot to unseen environments. We show that SORT3D achieves state-of-the-art zero-shot performance on complex view-dependent grounding tasks on two benchmarks. We also implement the pipeline to run real-time on two autonomous vehicles and demonstrate that our approach can be used for object-goal navigation on previously unseen real-world environments. All source code for the system pipeline is publicly released at https://github.com/nzantout/SORT3D.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。