arXiv:2411.14594cs.CV2024-11被引 8

将零样本3D视觉定位转化为约束满足问题,提升定位准确率。

Solving Zero-Shot 3D Visual Grounding as Constraint Satisfaction Problems

  • 把3D物体与空间关系建模为变量和约束,实现全局符号推理。
  • 在ScanRefer和Nr3D上分别提升7.0%和11.2%的准确率。
  • 支持否定和计数类查询,适合需要强语言理解的场景。

3D视觉定位旨在根据自然语言描述定位3D场景中的物体。监督方法虽精度尚可,但词汇封闭且语言理解能力有限。零样本方法利用大语言模型(LLMs)处理自然语言,或直接生成定位结果,或生成计算结果的程序(符号化)。本文提出一种零样本方法,将3DVG任务重新表述为约束满足问题(CSP),其中变量代表物体,约束代表其空间关系。该方法实现所有相关物体的全局符号推理,同时输出目标与锚定物体的定位结果。此外,我们展示了框架的灵活性:仅需少量额外编码即可处理否定和计数类查询。所提出的系统CSVG在公开数据集ScanRefer和Nr3D上使用开源LLM进行评估。结果表明,CSVG有效且优于当前最先进的零样本3DVG方法,在ScanRefer和Nr3D上的[email protected]得分分别提升了7.0%和11.2%。代码已公开于https://asig-x.github.io/csvg_web。

原文摘要 · Abstract (English)

3D visual grounding (3DVG) aims to locate objects in a 3D scene with natural language descriptions. Supervised methods have achieved decent accuracy, but have a closed vocabulary and limited language understanding ability. Zero-shot methods utilize large language models (LLMs) to handle natural language descriptions, where the LLM either produces grounding results directly or generates programs that compute results (symbolically). In this work, we propose a zero-shot method that reformulates the 3DVG task as a Constraint Satisfaction Problem (CSP), where the variables and constraints represent objects and their spatial relations, respectively. This allows a global symbolic reasoning of all relevant objects, producing grounding results of both the target and anchor objects. Moreover, we demonstrate the flexibility of our framework by handling negation- and counting-based queries with only minor extra coding efforts. Our system, Constraint Satisfaction Visual Grounding (CSVG), has been extensively evaluated on the public datasets ScanRefer and Nr3D datasets using only open-source LLMs. Results show the effectiveness of CSVG and superior grounding accuracy over current state-of-the-art zero-shot 3DVG methods with improvements of $+7.0\%$ ([email protected] score) and $+11.2\%$ on the ScanRefer and Nr3D datasets, respectively. The code of our system is available at https://asig-x.github.io/csvg_web.

3D视觉定位大语言模型符号推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。