让视频中的物体保持时间一致,提升视觉模型的推理能力。
Object-Centric Temporal Consistency via Conditional Autoregressive Inductive Biases
- 用自回归先验和一致性损失,让物体槽位跨帧稳定对应。
- 在视频预测和视觉问答任务中优于基线方法。
- 适合需要长期时序一致性的视频理解场景。
从视频中无监督学习物体中心表征是一种有前景的方法,可获得可用于多种下游任务(如预测与推理)的组合式表示。最近研究表明,预训练的视觉变换器(ViTs)可在真实世界视频数据集上有效学习物体中心表征。然而,尽管这些方法能成功提取场景中的物体,基于槽位的表征在连续帧间缺乏时间一致性,即物体到槽位的映射随视频变化。为此,我们提出条件自回归槽注意力(CA-SA),一种增强视频中心视觉任务中物体中心表征时间一致性的框架。通过利用自回归先验网络将表征条件化于先前时间步,并引入新颖的一致性损失函数,CA-SA 预测未来槽位表征并强制跨帧一致性。我们展示了定性和定量结果,表明所提方法在视频预测和视觉问答等下游任务中优于对比基线。
原文摘要 · Abstract (English)
Unsupervised object-centric learning from videos is a promising approach towards learning compositional representations that can be applied to various downstream tasks, such as prediction and reasoning. Recently, it was shown that pretrained Vision Transformers (ViTs) can be useful to learn object-centric representations on real-world video datasets. However, while these approaches succeed at extracting objects from the scenes, the slot-based representations fail to maintain temporal consistency across consecutive frames in a video, i.e. the mapping of objects to slots changes across the video. To address this, we introduce Conditional Autoregressive Slot Attention (CA-SA), a framework that enhances the temporal consistency of extracted object-centric representations in video-centric vision tasks. Leveraging an autoregressive prior network to condition representations on previous timesteps and a novel consistency loss function, CA-SA predicts future slot representations and imposes consistency across frames. We present qualitative and quantitative results showing that our proposed method outperforms the considered baselines on downstream tasks, such as video prediction and visual question-answering tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。