实时检测操作错误并用大模型生成解释，提升人机协作效率
Technical Report for Egocentric Mistake Detection for the HoloAssist Challenge
- 结合在线分析与大语言模型，同时识别流程与执行错误
- 在HoloAssist基准上取得第二名，有效识别多种错误类型
- 适合工业培训与智能辅助系统，支持即时纠错反馈
本文针对工业自动化与教育领域中实时视频分析的关键需求，提出一种在线错误检测框架。该框架不仅关注动作顺序等流程性错误，还处理电机滑移、工具误用等执行错误。一旦检测到错误，利用大语言模型（LLM）生成解释性反馈。在HoloAssist基准上的实验表明，该方法在错误检测任务中位列第二，验证了其有效性与实用性。
原文摘要 · Abstract (English)
In this report, we address the task of online mistake detection, which is vital in domains like industrial automation and education, where real-time video analysis allows human operators to correct errors as they occur. While previous work focuses on procedural errors involving action order, broader error types must be addressed for real-world use. We introduce an online mistake detection framework that handles both procedural and execution errors (e.g., motor slips or tool misuse). Upon detecting an error, we use a large language model (LLM) to generate explanatory feedback. Experiments on the HoloAssist benchmark confirm the effectiveness of our approach, where our approach is placed second on the mistake detection task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。