PRISM通过分层诊断框架,揭示大模型在虚拟家居任务中的失败根源。
PRISM: : Planning and Reasoning with Intent in Simulated Embodied Environments

- 构建三层次能力测试:基础感知、隐含意图理解、长期任务规划
- 发现轻量模型长程任务成功率仅20.0%,但消耗更多token
- 支持任意智能体评估与组件级故障分析,适合系统开发者
当基于大语言模型的具身智能体在家庭任务中失败时,原因可能是物体识别错误、遗忘子目标或动作序列不当——但现有基准仅报告单一成功率,无法定位问题所在。我们提出PRISM,一种诊断性基准,将问题从‘是否成功’重构为‘哪个能力最可能导致失败’。基于五个逼真多房间公寓(每栋4–8间房),PRISM将300项人工验证的任务划分为三个能力层级:基础能力(感知到行动的对齐)、推理能力(隐含意图解析)和长程能力(持续多步协调)。该基准提供不依赖智能体的可执行动作API,支持各类智能体(如LLM、VLM、符号规划器、强化学习策略及混合系统)在统一协议下端到端评估。为支持深入诊断,可选的感知、记忆、规划探针可被采用、替换或完全跳过,实现受控的组件级分析。七种主流LLM的实验表明清晰的能力层级:在理想感知条件下,空间对齐并非主要失败原因;所有模型家族均受隐含意图解析制约;长程协调暴露显著能力断崖——轻量模型成功率低至20.0%,且比前沿模型消耗更多token,体现补偿性过度推理而非真实规划能力。
原文摘要 · Abstract (English)
When an LLM-based embodied agent fails at a household task, the culprit could be misidentified objects, forgotten sub-goals, or poor action sequencing -- yet existing benchmarks report only a single success rate, making it impossible to tell which cognitive module is responsible. We present PRISM, a diagnostic benchmark that reframes this problem: rather than asking only \textit{did the agent succeed?}, PRISM asks \textit{which capability is most likely responsible for failure?} Built on five photorealistic multi-room apartments (4--8 rooms each), PRISM structures 300 human-verified tasks into three capability tiers -- \textit{Basic Ability}, \textit{Reasoning Ability}, and \textit{Long-horizon Ability} -- that isolate perception-to-action grounding, implicit intent resolution, and sustained multi-step coordination respectively. PRISM exposes an agent-agnostic executable action API that allows arbitrary agents: LLM agents, VLM agents, symbolic planners, RL policies, and hybrid systems, to be evaluated end-to-end under the same benchmark protocol. To support deeper diagnosis, optional probes for perception, memory, and planning can be adopted, replaced, or bypassed entirely, enabling controlled component-level analysis when desired. Experiments on seven contemporary LLMs establish a clear hierarchy: explicit spatial grounding is not the dominant failure source under oracle perception, implicit intent resolution is a significant bottleneck for all model families, and long-horizon coordination exposes a stark capability cliff -- lightweight models collapse to as low as 20.0\% success while simultaneously consuming more tokens than their frontier counterparts, a signature of compensatory over-reasoning rather than genuine planning capability. Project page: \href{https://sj-li.com/PROJ/PRISM}{link}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。