提出时间槽激活机制,让视频对象表示在遮挡时保持稳定,不乱改也不干扰重建。
TSA: Temporal Slot Activation for Persistent Object-Centric Video Representation

- 为每个槽设计动态激活分数,决定是否更新或参与重建
- 在遮挡时减少状态漂移,提升长期追踪准确率
- 适合需要稳定对象表示的视频理解任务,如长视频跟踪
无监督视频对象中心学习旨在将动态场景分解为具有时间持续性的实体表示。现有循环视频槽注意力方法在帧间传播固定数量的槽,但通常假设无条件传播:每个槽在每帧都更新和解码,无论其对应物体是否可见。我们指出,这种设计违背了持久槽的基本生命周期要求:当物体缺失或完全遮挡时,其槽应保持先前状态,并避免解释无关可见内容。无条件传播导致两种失效路径:更新引起的状态漂移,即当前帧信息覆盖缺失物体的表示;解码器引发的重建干扰,即无效槽仍通过解码器注意力与重建耦合。本文提出时间槽激活(TSA)机制,学习每个槽在每帧的激活分数 α_{k,t} ∈ (0,1),无需可见性标注。该激活分数作为共享潜在控制变量,用于建模槽的生命周期。当槽未激活时,TSA通过激活门控更新锚定其状态至前一帧,并通过激活相关加性偏置抑制其解码器参与(作用于softmax前的注意力对数)。这联合减少了状态漂移与重建驱动的干扰。为进一步提升部分遮挡和渐进重现下的决策能力,TSA还基于每个槽的时序记忆(由时序上下文编码器生成)来条件化激活预测。我们在MOVi-C/E、YT-VIS和OVIS基准上使用标准与基于追踪的指标(FG-ARI、mBO、IDF1、HOTA)评估TSA。结果表明,TSA在对象分解和时间身份保持方面持续提升,尤其在长时、高度遮挡视频中表现显著增强。
原文摘要 · Abstract (English)
Unsupervised video object-centric learning aims to decompose dynamic scenes into temporally persistent entity representations. Existing recurrent video slot-attention methods propagate a fixed set of slots across frames, but typically assume unconditional slot propagation: every slot is updated and decoded at every frame, regardless of whether its corresponding object is visible. We show that this design violates a basic lifecycle requirement for persistent slots: when an object is absent or fully occluded, its slot should preserve its previous state and avoid explaining unrelated visible content. Instead, unconditional propagation creates two failure pathways: update-induced state drift, where current-frame evidence overwrites the absent object's representation, and decoder-induced reconstruction interference, where the inactive slot remains coupled to reconstruction through decoder attention. We propose Temporal Slot Activation (TSA), a mechanism that learns a per-slot, per-frame activation score $α_{k,t} \in (0, 1)$ without visibility supervision. TSA uses this activation as a shared latent control variable for slot lifecycle modeling. When a slot is inactive, TSA anchors its state to the previous slot via activation-gated updating and suppresses its decoder participation through an activation-dependent additive bias on attention logits before softmax normalization. This jointly reduces state drift and reconstruction-driven interference. To improve decisions under partial occlusion and gradual reappearance, TSA further conditions activation prediction on a per-slot temporal memory produced by a Temporal Context Encoder. We evaluate TSA on MOVi-C/E, YT-VIS, and OVIS benchmarks using both standard and tracking-based metrics (FG-ARI, mBO, IDF1, HOTA). TSA consistently improves object decomposition and temporal identity preservation, with large gains on long, heavily occluded videos.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。