通过随机配对槽特征预测视频注意力查询，提升对象级场景理解性能。
Predicting Video Slot Attention Queries from Random Slot-Feature Pairs
- 用随机槽-特征对训练过渡器，融合多源信息增强查询预测
- 在对象发现任务上最高提升10分，达到新基准
- 适合研究视频对象中心学习与自监督表征的学者
无监督视频对象中心学习（OCL）具有潜力，可实现类人级的对象级场景表征与理解。主流视频OCL方法采用递归架构：聚合器将当前帧聚合为称为槽的对象特征，需依赖查询；过渡器将当前槽转换为下一帧的查询。该架构虽有效，但现有方法均存在两大缺陷：（i1）忽略下一帧特征——查询预测中最关键的信息源；（i2）未能学习过渡动态，而这是查询预测的核心知识。为此，我们提出随机槽-特征对用于查询预测（RandSF.Q）：（t1）设计新过渡器，同时融合槽与特征，提供更多预测信息；（t2）在可用递归中随机采样槽-特征对训练过渡器，使其学习过渡动态。在场景表征实验中，本方法显著超越现有视频OCL方法，例如对象发现任务最高提升10分，创下新基准。该优势也惠及下游任务如场景理解。源代码、模型检查点、训练日志：https://github.com/Genera1Z/RandSF.Q
原文摘要 · Abstract (English)
Unsupervised video Object-Centric Learning (OCL) is promising as it enables object-level scene representation and understanding as we humans do. Mainstream video OCL methods adopt a recurrent architecture: An aggregator aggregates current video frame into object features, termed slots, under some queries; A transitioner transits current slots to queries for the next frame. This is an effective architecture but all existing implementations both (\textit{i1}) neglect to incorporate next frame features, the most informative source for query prediction, and (\textit{i2}) fail to learn transition dynamics, the knowledge essential for query prediction. To address these issues, we propose Random Slot-Feature pair for learning Query prediction (RandSF.Q): (\textit{t1}) We design a new transitioner to incorporate both slots and features, which provides more information for query prediction; (\textit{t2}) We train the transitioner to predict queries from slot-feature pairs randomly sampled from available recurrences, which drives it to learn transition dynamics. Experiments on scene representation demonstrate that our method surpass existing video OCL methods significantly, e.g., up to 10 points on object discovery, setting new state-of-the-art. Such superiority also benefits downstream tasks like scene understanding. Source Code, Model Checkpoints, Training Logs: https://github.com/Genera1Z/RandSF.Q
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。