通过运动模式对比学习，提升视频场景图生成的时序关系理解能力。
Motion-aware Contrastive Learning for Temporal Panoptic Scene Graph Generation
- 基于运动模式设计对比学习框架，增强时序实体关联表示。
- 在视频与4D数据集上显著超越现有方法，提升场景图生成精度。
- 适合研究视频理解、时序关系建模及多模态场景分析的开发者。
为使人工智能具备对时序世界的全面理解，视频与4D全景场景图生成将视觉数据抽象为节点表示实体，边表示时序关系。现有方法通过跨时间维度追踪的实体掩码（掩码管）编码，再使用时序池化预测关系，但未能充分挖掘实体间关系所蕴含的运动信息。为此，本文提出一种面向运动模式的对比学习框架，用于时序场景图生成。首先，框架促使相似的主-关-宾三元组掩码管学习相近表示；其次，推动同一掩码管与其时间打乱版本间的表示分离；此外，还学习同一视频中不同三元组掩码管之间的远距离表示。大量实验表明，该运动感知对比框架在视频与4D数据集上显著优于现有先进方法。代码已开源：https://github.com/nguyentthong/motion-contrastive-sgg。
原文摘要 · Abstract (English)
To equip artificial intelligence with a comprehensive understanding towards a temporal world, video and 4D panoptic scene graph generation abstracts visual data into nodes to represent entities and edges to capture temporal relations. Existing methods encode entity masks tracked across temporal dimensions (mask tubes), then predict their relations with temporal pooling operation, which does not fully utilize the motion indicative of the entities' relation. To overcome this limitation, we introduce a contrastive representation learning framework that focuses on motion pattern for temporal scene graph generation. Firstly, our framework encourages the model to learn close representations for mask tubes of similar subject-relation-object triplets. Secondly, we seek to push apart mask tubes from their temporally shuffled versions. Moreover, we also learn distant representations for mask tubes belonging to the same video but different triplets. Extensive experiments show that our motion-aware contrastive framework significantly improves state-of-the-art methods on both video and 4D datasets. Code is available at: https://github.com/nguyentthong/motion-contrastive-sgg
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。