用LLM生成状态变化描述,让模型学会想象错误场景的因果关系。
What Changed and What Could Have Changed? State-Change Counterfactuals for Procedure-Aware Video Representation Learning
- 用大模型生成动作导致的状态变化作为监督信号
- 通过反事实场景训练,提升对错误步骤的识别能力
- 适合需要理解流程因果的视频分析任务
理解程序性活动需同时建模动作如何改变场景,以及场景变化如何影响后续动作序列,包括意外或错误行为。现有研究关注动作的时间顺序,但未显式学习状态变化(场景变换)。本文通过大语言模型(LLM)生成的状态变化描述作为视频编码器的监督信号,引入状态变化反事实(counterfactuals),模拟假设的失败结果,使模型能通过想象“如果……会怎样”的情景来学习。这种反事实推理增强了模型对每一步因果关系的理解。我们在时间动作分割、错误检测、动作阶段分类、帧检索、多实例检索和动作识别等任务上进行大量实验,结果表明所提状态变化描述及其反事实机制在多个任务上均取得显著性能提升。
原文摘要 · Abstract (English)
Understanding a procedural activity requires modeling both how action steps transform the scene, and how evolving scene transformations can influence the sequence of action steps, even those that are accidental or erroneous. Existing work has studied procedure-aware video representations by modeling the temporal order of actions, but has not explicitly learned the state changes (scene transformations). In this work, we study procedure-aware video representation learning by incorporating state-change descriptions generated by Large Language Models (LLMs) as supervision signals for video encoders. Moreover, we generate state-change counterfactuals that simulate hypothesized failure outcomes, allowing models to learn by imagining unseen "What if" scenarios. This counterfactual reasoning facilitates the model's ability to understand the cause and effect of each step in an activity. We conduct extensive experiments on procedure-aware tasks, including temporal action segmentation, error detection, action phase classification, frame retrieval, multi-instance retrieval, and action recognition. Our results demonstrate the effectiveness of the proposed state-change descriptions and their counterfactuals, and achieve significant improvements on multiple tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。