让智能体主动探索空间,突破被动感知局限。
ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loop

- 将观察者变为行动者,通过主动探索获取空间信息
- 主动探索比被动感知性能显著提升,随机多视角反而引入噪声
- 揭示模型存在动作盲区与元认知缺陷,适合研究具身智能的学者
空间智能依赖感知-行动闭环:智能体通过行动获取观测,并推理观测如何随行为变化。不同于被动感知,它们主动揭示遮挡结构、动态关系、包含性及功能等无法仅靠静态观察解析的信息。我们摒弃以往假设理想观测的设定,将观察者重构为行动者。提出 ESI-BENCH,一个基于 OmniGibson 的综合性具身空间智能基准,涵盖 10 大任务类别与 29 个子类,根植于 Spelke 的核心知识系统。智能体需决策部署感知、移动与操作能力,并合理规划顺序以主动积累任务相关证据。对前沿多模态大模型的实验表明,主动探索显著优于被动方法,智能体自发发现空间策略而无需显式指令;而随机多视角虽消耗更多图像,却常引入噪声而非有效信号。多数失败源于‘动作盲区’:错误动作导致劣质观测,引发连锁错误。显式 3D 建模虽稳定深度敏感任务表现,但不完美的 3D 表示反而因扭曲空间关系造成更大损害。人类研究表明,人会主动寻找反例并修正信念,而模型则在证据质量不足时仍高自信地过早下结论,暴露元认知缺陷——仅提升感知或增加具身交互无法解决此问题。
原文摘要 · Abstract (English)
Spatial intelligence unfolds through a perception-action loop: agents act to acquire observations, and reason about how observations vary as a function of action. Rather than passively processing what is seen, they actively uncover what is unseen - occluded structure, dynamics, containment, and functionality that cannot be resolved from passive sensing alone. We move beyond prior formulations of spatial intelligence that assume oracle observations by recasting the observer as an actor. We introduce ESI-BENCH, a comprehensive benchmark for embodied spatial intelligence spanning 10 task categories and 29 subcategories built on OmniGibson, grounded in Spelke's core knowledge systems. Agents must decide what abilities to deploy - perception, locomotion, and manipulation - and how to sequence them to actively accumulate task-relevant evidence. We conduct extensive experiments on state-of-the-art MLLMs and find that active exploration substantially outperforms passive counterparts, with agents spontaneously discovering emergent spatial strategies without explicit instructions, while random multi-view often adds noise rather than signal despite consuming far more images. Most failures stem not from weak perception but from action blindness: poor action choices lead to poor observations, which in turn drive cascading errors. While explicit 3D grounding stabilizes reasoning on depth-sensitive tasks, imperfect 3D representation proves more harmful than 2D baselines by distorting spatial relations. Human studies further reveal that unlike humans who seek falsifying viewpoints and revise beliefs under contradiction, models commit prematurely with high confidence regardless of evidence quality, exposing a metacognitive gap that neither better perception nor more embodied interaction alone can close.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。