arXiv:2608.14835cs.CV2026-08

首个端到端开放词汇动态场景图生成框架,支持从未见过的物体和关系。

OvDSGG: End-to-End Open-Vocabulary Dynamic Scene Graph Generation

论文配图:OvDSGG: End-to-End Open-Vocabulary Dynamic Scene Graph Generation
图 1 · 摘自论文原文
  • 基于视觉-语言对齐机制,实现无需知识蒸馏的开放词汇识别。
  • 在新概念上零样本召回率提升10.0至20.4个百分点,优于现有方法。
  • 适用于视频理解、问答等任务,适合研究开放世界视觉认知的学者。

动态场景图(DSG)以⟨主体, 谓词, 客体⟩三元组形式捕捉视频中的时空交互,支撑视频描述、视频问答与动作分析等下游任务。然而,现有端到端动态场景图生成(DSGG)方法为封闭集:仅能识别训练词汇表中的物体与谓词,难以处理稀有概念的长尾分布,严重限制实际应用。现有开放词汇模型通常依赖预训练大语言模型,导致多阶段训练与推理,开销巨大。本文提出OvDSGG,首个面向开放词汇的端到端DSGG框架。OvDSGG基于开放词汇空间主干网络与时间主干网络,引入三元组特征提取模块连接二者,并设计视觉-语言对齐模块,在联合视觉-语言特征空间中学习自适应决策边界,实现开放词汇识别,无需昂贵的知识蒸馏。此外,我们构建了基于Action Genome的严格开放词汇DSGG基准,对物体与谓词均采用不相交的Base/Novel划分。OvDSGG在所有指标上显著超越开放词汇基线,零样本召回率@K提升10.0–20.4个百分点,同时在封闭集任务上保持与最先进模型相当的性能。代码与基准已开源。

原文摘要 · Abstract (English)

Dynamic scene graphs (DSGs) capture spatio-temporal interactions across videos as $\langle$subject, predicate, object$\rangle$ triplets, and underpin downstream tasks such as video captioning, video question answering, and action analysis. However, end-to-end dynamic scene graph generation (DSGG) methods are closed-set: they recognize only objects and predicates from a fixed training vocabulary and struggle with the long-tailed distribution of rare concepts, severely limiting their real-world applicability. Existing open-vocabulary models typically inherit pretrained large language models, resulting in multi-stage training and inference with substantial cost. We introduce OvDSGG, the first end-to-end framework for open-vocabulary DSGG. OvDSGG builds on top of an open-vocabulary Spatial Backbone and a Temporal Backbone; we further propose a Triplet Feature Extraction Module that bridges them, and a Visual-Language Alignment Module that preserves open-vocabulary recognition by learning an adaptive decision boundary in the joint visual-language feature space, without expensive knowledge distillation in existing methods. We further introduce a rigorous open-vocabulary DSGG benchmark adapted from Action Genome, with disjoint Base/Novel splits for both objects and predicates. OvDSGG significantly outperforms open-vocabulary baselines across all metrics, with zero-shot Recall@$K$ scores 10.0--20.4 percentage point higher than the next-best baseline, while on closed-set DSGG remaining competitive with state-of-the-art models. Code and benchmark are publicly available at https://github.com/jhelsby/OvDSGG/.

动态场景图开放词汇视频理解视觉语言对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。