首个基于动态场景图的细粒度视频片段检索模型,提升时间关系建模能力。
TBSG-Net: Temporal Bipartite Scene Graph Network for Fine-Grained Video Moment Retrieval

- 构建动态场景图捕捉物体随时间的交互变化
- 显式编码关系持续时间,定位精度显著提升
- 融合Transformer与图卷积,兼顾全局与局部推理
最近的无候选视频片段检索(VMR)进展表明静态场景图(SSGs)在增强检索相关视频表征方面具有有效性。然而,将SSGs用于VMR仍受限于两大固有缺陷:(1) 缺乏时间动态性,无法建模物体及其关系随时间演变的过程,导致视频表征中丢失关键的时间依赖;(2) 缺乏显式的时间跨度编码,难以精确定位事件持续时间。为此,我们提出时序二分场景图网络(TBSG-Net),据我们所知,这是首个基于动态场景图(DSG)的无候选VMR模型。TBSG-Net利用DSG提取以事件为中心的图表示,实现对物体交互随时间演化的建模,解决了上述第(1)个问题。随后,通过新颖的动态场景图嵌入(DSG-E)模块捕获时序跨度与时空信息:首先,采用TBSG构造器将DSG转换为时序二分场景图(TBSG),显式编码对象、关系及时间跨度,解决第(2)个问题;其次,将所得TBSG输入混合式TBSG编码器,该编码器结合一种Transformer变体进行全局事件建模,以及图卷积网络进行细致的关系推理,最终生成更全面的时空表征。实验表明,TBSG-Net在所有基线方法上均取得显著性能提升。
原文摘要 · Abstract (English)
Recent advances in proposal-free Video Moment Retrieval (VMR) have highlighted the effectiveness of Static Scene Graphs (SSGs). By modeling objects and their relations at the frame level, SSGs enrich retrieval-oriented video representations. However, integrating SSGs into VMR remains constrained by two inherent limitations: (1) Lack of Temporal Dynamics. SSGs fail to model how objects and their relationships evolve over time, leading to the loss of essential temporal dependencies in video representation; and (2) Lack of Explicit Temporal Span Encoding. SSGs do not explicitly encode the duration of relationships, making precise localization challenging. To address these limitations, we propose Temporal Bipartite Scene Graph Network (TBSG-Net)---to the best of our knowledge, the first Dynamic Scene Graph (DSG) based proposal-free VMR model. Specifically, TBSG-Net leverages DSGs to extract event-centric graph representations of the input video, enabling the modeling of object interactions over time and thus addressing limitation (1). These DSGs are then processed by a novel Dynamic Scene Graph Embedding (DSG-E) module to capture both Temporal Span and spatio-temporal information. First, DSG-E utilizes a TBSG Constructor to transform DSGs into TBSGs, explicitly encoding objects, relationships, and time spans to tackle limitation (2). Second, the resultant TBSGs are passed into a hybrid TBSG Encoder that integrates a Transformer variant for global event modeling and a Graph Convolutional Network for detailed relational reasoning, ultimately producing a more comprehensive spatio-temporal representation. Our experiments demonstrate substantial improvements of TBSG-Net over all baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。