用大模型生成状态变化描述,让视频模型学会理解操作的因果关系。
EgoVIS@CVPR: What Changed and What Could Have Changed? State-Change Counterfactuals for Procedure-Aware Video Representation Learning
- 用大模型生成动作导致的状态变化作为监督信号
- 通过反事实场景模拟失败结果,提升对错误的识别能力
- 适合需要理解操作逻辑的视频分析任务
理解流程性活动需同时建模动作步骤如何改变场景,以及场景变化如何影响后续动作序列,包括意外或错误行为。现有流程感知视频表示方法未能显式学习状态变化(场景变换)。本文通过引入大语言模型生成的状态变化描述作为视频编码器的监督信号,实现流程感知视频表征学习。此外,我们生成状态变化反事实场景,模拟假设性失败结果,使模型通过想象“如果……会怎样”的未见情景进行学习。这种反事实推理增强了模型对每个步骤因果关系的理解。我们在时间动作分割、错误检测等流程感知任务上进行了大量实验,结果表明所提状态变化描述及其反事实机制有效,显著提升了多个任务性能。
原文摘要 · Abstract (English)
Understanding a procedural activity requires modeling both how action steps transform the scene, and how evolving scene transformations can influence the sequence of action steps, even those that are accidental or erroneous. Yet, existing work on procedure-aware video representations fails to explicitly learned the state changes (scene transformations). In this work, we study procedure-aware video representation learning by incorporating state-change descriptions generated by LLMs as supervision signals for video encoders. Moreover, we generate state-change counterfactuals that simulate hypothesized failure outcomes, allowing models to learn by imagining the unseen ``What if'' scenarios. This counterfactual reasoning facilitates the model's ability to understand the cause and effect of each step in an activity. To verify the procedure awareness of our model, we conduct extensive experiments on procedure-aware tasks, including temporal action segmentation, error detection, and more. Our results demonstrate the effectiveness of the proposed state-change descriptions and their counterfactuals, and achieve significant improvements on multiple tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。