提出新评估框架VIGIL,区分智能体是否真正完成任务并正确终止。
Done, But Not Sure: Disentangling World Completion from Self-Termination in Embodied Agents

- 设计VIGIL框架,通过语义报告独立测量任务完成与终止行为。
- 20个模型在1000次测试中,完成率相近但终止正确率差达19.7个百分点。
- 适用于研究智能体任务决策与行为终止能力的学者。
标准具身评估无法独立衡量智能体在回合结束时是否正确地做出任务完成决策,这种能力我们称为终端承诺。行为上不同的失败——从未完成任务、完成任务但未停止、无充分证据即报告成功——均被归为同一类基准失败。为此,我们提出VIGIL评估框架,使终端承诺可独立测量。在默认协议下,智能体仅接收自我中心RGB视觉输入,不获动作成功反馈,必须以语义报告结束每轮,该报告由隐藏世界状态严格验证。此设计产生两个独立得分:世界状态完成(W)和基准成功(B),后者还需正确的终端报告。该解耦使四种结果可区分:任务遗漏执行、达成后漂移、无依据承诺、验证成功。在1000个冻结回合中,20个模型的完成率相近,但基准成功率相差高达19.7个百分点:一个模型能将达成状态转化为正确报告,另一个虽执行接近却在目标后持续漂移而不关闭。动作-反馈干预进一步验证分离效果:以执行为导向的信号普遍提升W,但未在达成状态中建立终端报告的模型仍持续出现承诺失败。VIGIL提供一种使终端承诺可见且可评分的协议。
原文摘要 · Abstract (English)
Standard embodied evaluations do not independently score whether an agent correctly commits to task completion at episode closure, a capacity we call terminal commitment. Behaviorally distinct failures--never completing the task, completing it but failing to stop, and reporting success without sufficient evidence--collapse into the same benchmark failure. We introduce VIGIL, an evaluation framework that makes terminal commitment independently measurable. Under VIGIL's default protocol, agents observe only egocentric RGB, receive no action-success signals, and must end each episode with a semantic report checked deterministically against hidden world state. This yields two separate scores: world-state completion (W) and benchmark success (B), where B additionally requires a correct terminal report. This decoupling makes four outcome categories distinguishable: missed execution, post-attainment drift, unsupported commitment, and verified success. Across 20 models on 1,000 frozen episodes, systems with comparable W differ by up to 19.7 pp in B: one model converts achieved states into correct reports, while another with near-identical execution drifts past the goal without closing. An action-feedback intervention further tests the separation: execution-oriented signals improve W broadly, yet commitment failures persist in models that do not already ground terminal reports in the achieved state. VIGIL provides a protocol that makes terminal commitment independently visible and scorable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。