arXiv:2511.20073cs.CV2025-11中稿 · AAAI

用物体状态建模视频,让智能体理解复杂任务的执行过程。

Learning Procedural-aware Video Representations through State-Grounded Hierarchy Unfolding

  • 引入可观察的物体状态作为视觉对齐的语义层,构建任务-步骤-状态层级结构。
  • 在COIN和CrossTask数据集上,任务识别、步骤识别等任务性能显著提升。
  • 逐步预训练策略有效强化层次结构,适合需要推理与执行的智能体系统。

学习具备程序感知能力的视频表示是构建能推理并执行复杂任务智能体的关键。现有方法通常通过将视觉内容与任务及步骤级的文本描述对齐来注入程序语义,但任务和步骤描述过于抽象,难以与视觉数据中的具体细节建立可靠对齐。为此,我们引入“状态”——即物体配置的文本快照——作为与视觉数据直接对应的语义层,将抽象程序锚定在模型可感知的内容上。我们提出新的任务-步骤-状态(TSS)框架,其中任务通过步骤驱动状态间的转移实现。为强制该结构,我们设计一种渐进式预训练策略,逐步展开TSS层级,迫使模型在状态层面建立表示,并关联步骤与高层任务。在COIN和CrossTask数据集上的大量实验表明,该方法在任务识别、步骤识别和下一步预测等多个下游任务中均优于基线模型。消融实验显示,状态监督是各项任务性能提升的关键因素。此外,渐进式预训练比标准联合训练更有效,能更好维持预期的层次结构。

原文摘要 · Abstract (English)

Learning procedural-aware video representations is a key step towards building agents that can reason about and execute complex tasks. Existing methods typically address this problem by aligning visual content with textual descriptions at the task and step levels to inject procedural semantics into video representations. However, due to their high level of abstraction, 'task' and 'step' descriptions fail to form a robust alignment with the concrete, observable details in visual data. To address this, we introduce 'states', i.e., textual snapshots of object configurations, as a visually-grounded semantic layer that anchors abstract procedures to what a model can actually see. We formalize this insight in a novel Task-Step-State (TSS) framework, where tasks are achieved via steps that drive transitions between observable states. To enforce this structure, we propose a progressive pre-training strategy that unfolds the TSS hierarchy, forcing the model to ground representations in states while associating them with steps and high-level tasks. Extensive experiments on the COIN and CrossTask datasets show that our method outperforms baseline models on multiple downstream tasks, including task recognition, step recognition, and next step prediction. Ablation studies show that introducing state supervision is a key driver of performance gains across all tasks. Additionally, our progressive pretraining strategy proves more effective than standard joint training, as it better enforces the intended hierarchical structure.

视频理解程序推理多层级建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。