构建交互式物理推理基准，测试模型从感知到行动的连贯能力
From Perception to Action: An Interactive Benchmark for Vision Reasoning
- 设计3D物理驱动环境，评估模型对几何与因果关系的理解
- 顶尖模型在长序列动作规划中仍失败，难以将感知转化为可靠行动
- 适合研究具身智能、视觉语言模型与物理推理的学者参考
理解物理结构对具身智能体、交互设计和长时程操作等真实场景至关重要。然而，当前视觉语言模型（VLM）评估多采用无结构、单轮问答模式（如VQA），无法检验模型在动态环境中对几何、接触与支撑关系联合约束下可行动作的推理能力。为此，我们提出因果动作与交互层级（CHAIN）基准，一个交互式3D物理驱动测试平台，用于评估模型是否能基于物理约束进行理解、规划与执行结构化动作序列。该基准将评估范式从被动感知转向主动问题求解，涵盖互锁机械谜题、3D堆叠与包装等任务。我们在统一交互设置下对前沿VLM与基于扩散的模型进行了全面评测。结果表明，表现最优的模型仍难以内化物理结构与因果约束，常无法生成可靠的长时程计划，且无法稳健地将感知结构转化为有效动作。项目地址：https://social-ai-studio.github.io/CHAIN/
原文摘要 · Abstract (English)
Understanding the physical structure is essential for real-world applications such as embodied agents, interactive design, and long-horizon manipulation. Yet, prevailing Vision-Language Model (VLM) evaluations still center on structure-agnostic, single-turn setups (e.g., VQA), which fail to assess agents' ability to reason about how geometry, contact, and support relations jointly constrain what actions are possible in a dynamic environment. To address this gap, we introduce the Causal Hierarchy of Actions and Interactions (CHAIN) benchmark, an interactive 3D, physics-driven testbed designed to evaluate whether models can understand, plan, and execute structured action sequences grounded in physical constraints. CHAIN shifts evaluation from passive perception to active problem solving, spanning tasks such as interlocking mechanical puzzles and 3D stacking and packing. We conduct a comprehensive study of state-of-the-art VLMs and diffusion-based models under unified interactive settings. Our results show that top-performing models still struggle to internalize physical structure and causal constraints, often failing to produce reliable long-horizon plans and cannot robustly translate perceived structure into effective actions. The project is available at https://social-ai-studio.github.io/CHAIN/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。