用视频检测机器人执行任务时的时间错误，无需复杂标注。
TIMID: Time-Dependent Mistake Detection in Videos of Robot Executions
- 基于视频异常检测框架，输入视频和任务提示，输出逐帧错误判断。
- 仅需每段视频一个标签即可训练，支持弱监督学习。
- 自建带时序错误的仿真数据集，实现零样本跨仿真与真实场景测试。
随着机器人执行越来越复杂的任务序列，出错方式也日益增多。现有视频异常检测（VAD）方法多关注低层级运动或动作异常，难以识别不表现为低层错误的复杂时间或空间任务违规。本文提出新架构TIMID，专门检测机器人执行高层任务时的时间依赖性错误。该模型接收视频和任务/错误提示，输出帧级错误存在性预测。通过采用VAD范式，模型可仅用每视频一个标签进行弱监督训练。为缓解错误执行数据稀缺问题，我们构建了一个包含可控时序错误的多机器人仿真数据集，并融合真实执行数据，实现零样本仿真到真实迁移评估。实验表明，未经微调的视觉语言模型缺乏显式时序推理能力，而本框架能有效识别多种时序错误类型。
原文摘要 · Abstract (English)
As robotic systems execute increasingly difficult task sequences, so does the number of ways in which they can fail. Video Anomaly Detection (VAD) frameworks typically focus on singular, low-level kinematic or action failures, struggling to identify more complex temporal or spatial task violations, because they do not necessarily manifest as low-level execution errors. To address this problem, the main contribution of this paper is a new VAD-inspired architecture, TIMID, which is able to detect robot time-dependent mistakes when executing high-level tasks. Our architecture receives as inputs a video and prompts of the task and the potential mistake, and returns a frame-level prediction in the video of whether the mistake is present or not. By adopting a VAD formulation, the model can be trained with weak supervision, requiring only a single label per video. Additionally, to alleviate the problem of data scarcity of incorrect executions, we introduce a multi-robot simulation dataset with controlled temporal errors and real executions for zero-shot sim-to-real evaluation. Our experiments demonstrate that out-of-the-box VLMs lack the explicit temporal reasoning required for this task, whereas our framework successfully detects different types of temporal errors. Project: https://ropertunizar.github.io/TIMID/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。