让视频理解从看得到的物体扩展到看不见的物体,实现动态场景的全貌建模。
Towards Spatio-Temporal World Scene Graph Generation from Monocular Videos
- 构建4D场景图任务,融合时空信息与物体持久性假设
- 提出三种新方法,分别用缓存、掩码重建和时序注意力推理被遮挡物体
- 首个支持未观测对象关系预测的基准,适合视频理解与具身智能研究者
时空场景图为建模动态物体交互提供了一种系统化表示,但现有方法仍以帧为中心:仅关注当前可见物体,遮挡后即丢弃实体,并在2D空间中操作。为此,我们首次提出ActionGenome4D数据集,通过前馈3D重建将动作视频升级为4D场景,为每个参与动作的物体提供世界坐标系下的边界框,并标注密集关系,包括因遮挡或摄像机运动而暂时不可见的物体。基于此数据,我们正式定义世界场景图生成(WSGG)任务:在每个时间戳构建包含所有交互物体(无论是否可见)的世界场景图。我们提出三种互补方法:PWG(持久世界图),通过零阶特征缓存实现物体恒常性;MWAE(掩码世界自编码器),将未观测物体推理转化为跨视角关联检索的掩码补全;4DST(4D场景变换器),用可微分的物体级时序注意力替代静态缓存,结合3D运动与相机位姿特征。我们进一步设计并评估开源视觉-语言模型在WSGG任务上的表现,采用基于图RAG的方法建立未定位关系预测基线。WSGG推动视频场景理解向以世界为中心、时间持续且可解释的推理迈进。
原文摘要 · Abstract (English)
Spatio-temporal scene graphs provide a principled representation for modeling evolving object interactions, yet existing methods remain fundamentally frame-centric: they reason only about currently visible objects, discard entities upon occlusion, and operate in 2D. To address this, we first introduce ActionGenome4D, a dataset that upgrades Action Genome videos into 4D scenes via feed-forward 3D reconstruction, world-frame oriented bounding boxes for every object involved in actions, and dense relationship annotations including for objects that are temporarily unobserved due to occlusion or camera motion. Building on this data, we formalize World Scene Graph Generation (WSGG), the task of constructing a world scene graph at each timestamp that encompasses all interacting objects in the scene, both observed and unobserved. We then propose three complementary methods, each exploring a different inductive bias for reasoning about unobserved objects: PWG (Persistent World Graph), which implements object permanence via a zero-order feature buffer; MWAE (Masked World Auto-Encoder), which reframes unobserved-object reasoning as masked completion with cross-view associative retrieval; and 4DST (4D Scene Transformer), which replaces the static buffer with differentiable per-object temporal attention enriched by 3D motion and camera-pose features. We further design and evaluate the performance of strong open-source Vision-Language Models on the WSGG task via a suite of Graph RAG-based approaches, establishing baselines for unlocalized relationship prediction. WSGG thus advances video scene understanding toward world-centric, temporally persistent, and interpretable scene reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。