arXiv:2411.02570cs.CV2024-11被引 10

用双分支模型实时检测第一人称视频中的操作错误,无需事先知道错误类型。

TI-PREGO: Chain of Thought and In-Context Learning for Online Mistake Detection in PRocedural EGOcentric Videos

  • 通过动作识别与未来动作预测的对比发现错误
  • 在两个数据集上实现比现有方法更高的在线检测准确率
  • 适合制造业、医疗培训等需要实时纠错的场景

从第一人称视频中实时识别操作错误是一项关键但具有挑战性的任务,广泛应用于制造、医疗和技能训练等领域。由于错误类型可能未知或全新出现,系统必须具备开放集检测能力,不依赖失败样本。目前尚无有效方法能在线完成此类错误检测。本文提出一种双分支架构:一个分支持续识别当前动作,另一个分支基于识别结果预测未来动作。当实际动作与预测动作不一致时即判定为错误。识别分支处理输入帧,输出当前动作并生成动作标记;预测分支则利用大语言模型(LLM)的模式匹配能力,根据历史动作标记预测下一步。针对任务的在线特性,我们系统评估了逐帧判断的难点,尤其强调动态场景中准确及时预测的重要性。在两个程序性数据集上的大量实验表明,该方法能有效应对开放集错误检测挑战,展现出对多种变体和先进模型的鲁棒性与有效性。

原文摘要 · Abstract (English)

Identifying procedural errors online from egocentric videos is a critical yet challenging task across various domains, including manufacturing, healthcare, and skill-based training. The nature of such mistakes is inherently open-set, as unforeseen or novel errors may occur, necessitating robust detection systems that do not rely on prior examples of failure. Currently, however, no technique effectively detects open-set procedural mistakes online. We propose a dual branch architecture to address this problem in an online fashion: one branch continuously performs step recognition from the input egocentric video, while the other anticipates future steps based on the recognition module's output. Mistakes are detected as mismatches between the currently recognized action and the action predicted by the anticipation module. The recognition branch takes input frames, predicts the current action, and aggregates frame-level results into action tokens. The anticipation branch, specifically, leverages the solid pattern-matching capabilities of Large Language Models (LLMs) to predict action tokens based on previously predicted ones. Given the online nature of the task, we also thoroughly benchmark the difficulties associated with per-frame evaluations, particularly the need for accurate and timely predictions in dynamic online scenarios. Extensive experiments on two procedural datasets demonstrate the challenges and opportunities of leveraging a dual-branch architecture for mistake detection, showcasing the effectiveness of our proposed approach. In a thorough evaluation including recognition and anticipation variants and state-of-the-art models, our method reveals its robustness and effectiveness in online applications.

视频分析错误检测在线学习大模型应用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。