融合静态动态特征,提升视频定位精度。
Static and Dynamic Graph Alignment Network for Temporal Video Grounding

- 双图对齐:结合静态与动态特征构建互补图结构。
- 查询感知建模:通过对比学习实现图文精准对齐。
- 多粒度训练:渐进式学习提升边界精调能力。
时间视频定位(TVG)旨在从无剪辑视频中定位与自然语言查询语义对应的时序片段。近年来,图卷积网络(GCN)被广泛用于建模视频片段间的时序关系,通过构建片段级图增强上下文推理。然而,现有基于GCN的TVG方法存在三大瓶颈:1)多数方法仅使用静态或动态特征构建图节点,导致视觉表征不完整,忽略互补语义;2)多数方法以查询无关方式构建时序图,造成特征交互效率低下;3)通常采用单一粒度语义匹配,直接训练复杂定位任务易导致收敛慢、精度不足。为此,本文提出静态与动态图对齐网络(SDGAN)。首先,SDGAN联合利用静态与动态视觉特征构建两个互补时序图,并执行逐位置节点对齐,实现更丰富鲁棒的视觉表征。其次,引入查询-片段对比学习与自适应图建模,显式对齐视觉片段与对应文本查询,生成查询感知的视觉表示。第三,结合多粒度时序提议与渐进式由易到难训练策略,有效衔接粗粒度语义定位与细粒度时序边界优化。在三个基准数据集上的大量实验表明,SDGAN在复杂TVG场景下均取得优越性能。代码与数据集见https://github.com/ZhanJieHu/SDGAN。
原文摘要 · Abstract (English)
Temporal Video Grounding (TVG) aims to localize temporal moments in an untrimmed video that semantically correspond to given natural language queries. Recently, Graph Convolutional Networks (GCN) have been widely adopted in TVG to model temporal relations among video clips and enhance contextual reasoning by constructing clip-level graphs. Despite their effectiveness, existing GCN-based TVG methods encounter three critical bottlenecks: 1) Most methods construct graph nodes using either static or dynamic features alone, resulting in incomplete visual representation and overlooking complementary semantics, 2) Most methods construct temporal graphs in a query-agnostic manner, leading to inefficient feature interaction within the temporal graph representation, and 3) Most methods often suffer from a single-granularity semantic matching, while direct training on complex temporal localization task may lead to slow convergence and suboptimal precision. To address these challenges, we propose Static and Dynamic Graph Alignment Network (SDGAN). First, SDGAN jointly exploits static and dynamic visual features to construct two complementary temporal graphs and performs Position-wise Nodes Alignment, enabling more expressive and robust visual representation. Second, SDGAN introduces Query-Clip Contrastive Learning and Adaptive Graph Modeling to explicitly align visual clips with their corresponding textual queries, yielding query-aware visual representations. Third, SDGAN incorporates multi-granularity temporal proposals within Progressive Easy-to-Hard Training Strategy, effectively bridging coarse-grained semantic localization and fine-grained temporal boundary refinement. Extensive experiments on three benchmark datasets demonstrate that SDGAN achieves superior performance across complex TVG scenarios. Codes and datasets are available at https://github.com/ZhanJieHu/SDGAN.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。