arXiv:2511.10017cs.CV2025-11NeurIPS被引 16

让机器人通过指令精准识别3D场景中可操作物体的部位、位置和动作方式。

AffordBot: 3D Fine-grained Embodied Reasoning via Multimodal Large Language Models

  • 用多模态大模型结合思维链,从点云生成全景图并定位可操作区域。
  • 在SceneFun3D上达到最先进性能,仅需点云和语言指令即可推理。
  • 适合需要精细物理交互的智能机器人研究与应用。

物理环境中有效的人机协作不仅需要知道对什么进行操作,还需明确操作对象的位置及交互方式。现有方法通常在物体层面操作或割裂处理细粒度可操作性,缺乏连贯的指令驱动定位与推理。本文提出新任务:细粒度3D具身推理,要求智能体根据任务指令,为3D场景中每个被引用的可操作元素预测包含空间位置、运动类型和运动轴的结构化三元组。为此,我们提出AffordBot框架,将多模态大语言模型(MLLMs)与定制化的思维链(CoT)推理范式结合。为弥合3D输入与2D兼容的MLLM之间的差距,我们渲染场景的环视图像,并将3D元素候选投影到这些视图中,形成与场景几何一致的丰富视觉表示。我们的CoT流程首先通过主动感知阶段,引导MLLM根据指令选择最信息丰富的视角,再逐步推理以定位可操作元素并推断合理的交互动作。在SceneFun3D数据集上的评估表明,AffordBot实现了最先进性能,仅需3D点云输入与MLLM即展现出强泛化能力与物理上合理的推理。

原文摘要 · Abstract (English)

Effective human-agent collaboration in physical environments requires understanding not only what to act upon, but also where the actionable elements are and how to interact with them. Existing approaches often operate at the object level or disjointedly handle fine-grained affordance reasoning, lacking coherent, instruction-driven grounding and reasoning. In this work, we introduce a new task: Fine-grained 3D Embodied Reasoning, which requires an agent to predict, for each referenced affordance element in a 3D scene, a structured triplet comprising its spatial location, motion type, and motion axis, based on a task instruction. To solve this task, we propose AffordBot, a novel framework that integrates Multimodal Large Language Models (MLLMs) with a tailored chain-of-thought (CoT) reasoning paradigm. To bridge the gap between 3D input and 2D-compatible MLLMs, we render surround-view images of the scene and project 3D element candidates into these views, forming a rich visual representation aligned with the scene geometry. Our CoT pipeline begins with an active perception stage, prompting the MLLM to select the most informative viewpoint based on the instruction, before proceeding with step-by-step reasoning to localize affordance elements and infer plausible interaction motions. Evaluated on the SceneFun3D dataset, AffordBot achieves state-of-the-art performance, demonstrating strong generalization and physically grounded reasoning with only 3D point cloud input and MLLMs.

具身智能多模态推理3D理解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。