让AI通过动态生成代码接口，解决3D场景中的空间推理难题。
Visual Agentic AI for Spatial Reasoning with a Dynamic API
- 用多个大模型协作生成可动态扩展的Python API来拆解复杂视觉任务。
- 在新构建的3D多步推理基准上，性能超越现有零样本模型。
- 适合研究视觉智能体、3D理解与自动化程序合成的开发者参考。
视觉推理——理解视觉世界的能力——对在三维场景中运行的具身智能体至关重要。尽管当前人工智能在图像问答方面取得进展，但在3D空间推理任务中表现下降。为应对这一挑战，我们提出一种代理式程序合成方法：由大语言模型协作生成带有新函数的Python API，以解决常见子问题。该方法突破了依赖静态人工定义API的局限，可处理更广泛查询。为此，我们构建了一个包含多步定位与推断的新基准，用于评估AI在3D理解方面的能力。实验表明，我们的方法在3D视觉推理上优于现有零样本模型，并验证了该代理框架的有效性。项目主页：https://glab-caltech.github.io/vadar/
原文摘要 · Abstract (English)
Visual reasoning -- the ability to interpret the visual world -- is crucial for embodied agents that operate within three-dimensional scenes. Progress in AI has led to vision and language models capable of answering questions from images. However, their performance declines when tasked with 3D spatial reasoning. To tackle the complexity of such reasoning problems, we introduce an agentic program synthesis approach where LLM agents collaboratively generate a Pythonic API with new functions to solve common subproblems. Our method overcomes limitations of prior approaches that rely on a static, human-defined API, allowing it to handle a wider range of queries. To assess AI capabilities for 3D understanding, we introduce a new benchmark of queries involving multiple steps of grounding and inference. We show that our method outperforms prior zero-shot models for visual reasoning in 3D and empirically validate the effectiveness of our agentic framework for 3D spatial reasoning tasks. Project website: https://glab-caltech.github.io/vadar/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。