用工具增强多智能体，让大模型更准地安排3D物体。
VULCAN: Tool-Augmented Multi Agents for Iterative 3D Object Arrangement
- 用函数级API替代原始代码，提升语言指令与3D操作的对应精度。
- 引入视觉工具形成感知反馈环，实时验证空间操作效果。
- 多智能体分工协作，支持复杂指令迭代调整并自动纠错。
尽管多模态大语言模型在2D视觉-语言任务中取得显著进展，其在复杂3D场景操作中的应用仍处于探索阶段。本文针对3D物体排列任务中的三个关键挑战提出解决方案：首先，为克服MLLM弱视觉定位问题，引入基于MCP的API，将交互从脆弱的原始代码操作转为更稳健的函数级更新；其次，通过一系列专用视觉工具增强模型对3D场景的理解，实现对场景状态的分析、空间信息的获取及动作结果的验证，形成关键的感知反馈回路；第三，为应对迭代过程中的错误与不确定性，设计了具有规划、执行与验证角色的多智能体协作框架，有效分解任务流程并实现容错。我们在25个复杂物体排列任务上验证了该方法的有效性，显著优于现有基线。网站：vulcan-3d.github.io
原文摘要 · Abstract (English)
Despite the remarkable progress of Multimodal Large Language Models (MLLMs) in 2D vision-language tasks, their application to complex 3D scene manipulation remains underexplored. In this paper, we bridge this critical gap by tackling three key challenges in 3D object arrangement task using MLLMs. First, to address the weak visual grounding of MLLMs, which struggle to link programmatic edits with precise 3D outcomes, we introduce an MCP-based API. This shifts the interaction from brittle raw code manipulation to more robust, function-level updates. Second, we augment the MLLM's 3D scene understanding with a suite of specialized visual tools to analyze scene state, gather spatial information, and validate action outcomes. This perceptual feedback loop is critical for closing the gap between language-based updates and precise 3D-aware manipulation. Third, to manage the iterative, error-prone updates, we propose a collaborative multi-agent framework with designated roles for planning, execution, and verification. This decomposition allows the system to robustly handle multi-step instructions and recover from intermediate errors. We demonstrate the effectiveness of our approach on a diverse set of 25 complex object arrangement tasks, where it significantly outperforms existing baselines. Website: vulcan-3d.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。