预测视频中物体与关系的未来变化，实现动态场景图的完整生成。
FDSG: Forecasting Dynamic Scene Graphs
- 用查询分解与随机微分方程建模物体和关系的动态演化。
- 在Action Genome数据集上，未来场景图预测准确率显著领先现有方法。
- 适合做视频理解、未来行为预测的研究者参考。
动态场景图生成将场景图从图像拓展到视频，建模实体关系及其时序演变。然而，现有方法或仅基于观测帧生成场景图而未显式建模时序动态，或仅预测关系而假设实体标签和位置静态。这限制了对实体与关系演化的有效外推，制约视频场景理解。本文提出一种新框架FDSG，可预测未观测帧中物体的标签、边界框及关系，同时生成已观测帧的场景图。其场景图预测模块采用查询分解与神经随机微分方程建模动态演化，时序聚合模块通过交叉注意力融合预测与观测信息以优化结果。为评估该框架，我们引入新任务「场景图预测」。在Action Genome数据集上的实验表明，FDSG在动态场景图生成、场景图前瞻与场景图预测三项任务中均超越当前最优方法。代码将在发表后公开。
原文摘要 · Abstract (English)
Dynamic scene graph generation extends scene graph generation from images to videos by modeling entity relationships and their temporal evolution. However, existing methods either generate scene graphs from observed frames without explicitly modeling temporal dynamics, or predict only relationships while assuming static entity labels and locations. These limitations hinder effective extrapolation of both entity and relationship dynamics, restricting video scene understanding. We propose Forecasting Dynamic Scene Graphs (FDSG), a novel framework that predicts future entity labels, bounding boxes, and relationships, for unobserved frames, while also generating scene graphs for observed frames. Our scene graph forecast module leverages query decomposition and neural stochastic differential equations to model entity and relationship dynamics. A temporal aggregation module further refines predictions by integrating forecasted and observed information via cross-attention. To benchmark FDSG, we introduce Scene Graph Forecasting, a new task for full future scene graph prediction. Experiments on Action Genome show that FDSG outperforms state-of-the-art methods on dynamic scene graph generation, scene graph anticipation, and scene graph forecasting. Codes will be released upon publication.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。