用程序化规则检测智能体空间认知缺陷,发现顶级模型存在大量隐藏错误。
MetaSpace: Metamorphic Testing for Spatial Cognition in Embodied Agents

- 基于物理规律生成可验证的测试用例,自动判断空间推理是否合理。
- 在三个场景中检测出90,422处空间认知错误,顶尖模型得分仅0.44~0.52。
- 适合评估智能体安全性与可靠性,尤其关注导航、操作等具身任务。
具身智能体通过物理身体与环境交互。当前评估主要依赖人工标注的视觉问答对或高层任务完成度指标,前者耗时且质量不稳定,后者可能掩盖关键漏洞,使智能体通过非最优甚至危险方式完成任务。由于空间认知是执行具身任务的基础,亟需评估智能体在任务中是否具备稳健的空间认知能力。受软件工程中变形测试启发,我们提出MetaSpace框架,利用真实执行轨迹中的时空多模态状态,基于逻辑规则和物理定律自动生成测试用例。关键在于将这些规则编码为可执行的逻辑程序(Prolog)。违反这些规则即表明空间认知失败。在三个具身场景的实证评估中,MetaSpace成功检测到90,422个空间认知错误,针对最先进的多模态大模型驱动的智能体。我们引入空间认知(SC)分数量化表现,结果显示所有SOTA模型平均分介于0.44至0.52之间,显著低于人类基准0.96。
原文摘要 · Abstract (English)
An embodied agent is an intelligent entity that interacts with its environment through a physical body. Currently, the evaluation of embodied agents primarily relies on two paradigms: (1) manually annotated Visual Question Answering (VQA) pairs and (2) high-level task completion metrics, such as success in navigation or manipulation. The former is labor-intensive and subject to variability in annotation quality. The latter may obscure critical vulnerabilities, allowing agents to complete tasks through suboptimal means or safety violations, thereby concealing safety risks and inefficiencies. Given that spatial cognition is the cornerstone for executing embodied tasks, there is a pressing need to assess whether embodied agents possess robust spatial cognition during task execution. Inspired by metamorphic testing principles in software engineering, we propose MetaSpace, a novel framework designed to evaluate the spatial cognition of agents. By leveraging spatiotemporal multimodal states derived from real execution trajectories, MetaSpace automatically generates test cases based on predefined metamorphic relations (MRs) grounded in logical rules and physical laws. Crucially, we encode these MRs as executable rules in a logic programming language (Prolog). Violations of these relations indicate failures in spatial cognition. Our empirical evaluation across three embodied scenarios demonstrates that MetaSpace successfully detects 90,422 spatial cognition errors in state-of-the-art (SOTA) MLLM-driven agents. We introduce the Spatial Cognition (SC) score to quantify performance. Results indicate that all SOTA agents achieve average scores between 0.44 and 0.52, significantly lower than the human benchmark of 0.96.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。