arXiv:2512.03474cs.CV2025-12中稿 · ICLR被引 4

通过分析动作结果来检测流程任务中的错误,效果更可靠。

Procedural Mistake Detection via Action Effect Modeling

  • 用概率模型联合建模动作执行与结果,捕捉行为后果
  • 在两个基准上达到当前最佳,尤其在单类分类场景下
  • 适合需要精准纠错的智能学习系统或自动化任务指导

流程任务中的错误检测对构建支持学习和执行的智能系统至关重要。现有方法主要关注动作如何执行,却忽略了其产生的结果,即动作效应。然而,许多错误并不体现在动作本身,而体现在错误的对象状态或空间布局等结果上。为弥补这一空白,我们提出动作效应建模(AEM),一种通过概率形式统一捕捉动作执行及其结果的框架。AEM首先基于语义相关性和视觉质量,选择最具信息量的效果帧以识别动作结果;随后从视觉定位和符号场景图中提取互补线索,并将其对齐至共享潜在空间,形成鲁棒的效应感知表示。为实现错误检测,我们进一步设计了基于提示的检测器,结合任务特定提示,将每个动作片段与其预期执行语义对齐。该方法在EgoPER和CaptainCook4D基准上,在挑战性的单类分类(OCC)设置下取得当前最优性能。结果表明,同时建模执行与结果能带来更可靠的错误检测,并凸显了效应感知表示在更多下游应用中的潜力。

原文摘要 · Abstract (English)

Mistake detection in procedural tasks is essential for building intelligent systems that support learning and task execution. Existing approaches primarily analyze how an action is performed, while overlooking what it produces, i.e., the \textbf{action effect}. Yet many errors manifest not in the execution itself but in the resulting outcome, such as an unintended object state or incorrect spatial arrangement. To address this gap, we propose Action Effect Modeling (AEM), a unified framework that jointly captures action execution and its outcomes through a probabilistic formulation. AEM first identifies the outcome of an action by selecting the most informative effect frame based on semantic relevance and visual quality. It then extracts complementary cues from visual grounding and symbolic scene graphs, aligning them in a shared latent space to form robust effect-aware representations. To detect mistakes, we further design a prompt-based detector that incorporates task-specific prompts and aligns each action segment with its intended execution semantics. Our approach achieves state-of-the-art performance on the EgoPER and CaptainCook4D benchmarks under the challenging one-class classification (OCC) setting. These results demonstrate that modeling both execution and outcome yields more reliable mistake detection, and highlight the potential of effect-aware representations to benefit a broader range of downstream applications.

错误检测动作建模智能系统视觉理解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。