arXiv:2512.05524cs.CV2025-12被引 1

用视觉语言模型增强视频场景图生成,提升关系理解能力。

VOST-SGG: VLM-Aided One-Stage Spatio-Temporal Scene Graph Generation

  • 引入双源查询初始化,分离语义与位置信息,实现更精准的注意力机制。
  • 融合视觉、文本和空间线索构建多模态特征库,显著提升谓词分类准确率。
  • 适用于需要理解视频中对象动态关系的下游任务,如视频描述与问答。

时空场景图生成(ST-SGG)旨在建模视频帧间物体及其动态关系,为视频描述、视觉问答等下游任务提供可解释表示。尽管基于DETR的单阶段模型取得进展,仍存在两大瓶颈:一是依赖语义无关的可学习查询,初始化缺乏实例感知;二是仅使用单模态视觉特征进行谓词分类。为此,本文提出VOST-SGG框架,将视觉语言模型(VLM)的常识推理能力融入ST-SGG流程。首先,设计双源查询初始化策略,解耦“关注什么”与“关注哪里”,实现语义驱动的时空定位。其次,构建融合视觉、文本及空间线索的多模态特征库,用于提升谓词分类性能。在Action Genome数据集上的大量实验表明,该方法达到当前最优效果,验证了引入VLM语义先验与多模态特征的有效性。代码将开源于https://github.com/LUNAProject22/VOST。

原文摘要 · Abstract (English)

Spatio-temporal scene graph generation (ST-SGG) aims to model objects and their evolving relationships across video frames, enabling interpretable representations for downstream reasoning tasks such as video captioning and visual question answering. Despite recent advancements in DETR-style single-stage ST-SGG models, they still suffer from several key limitations. First, while these models rely on attention-based learnable queries as a core component, these learnable queries are semantically uninformed and instance-agnostically initialized. Second, these models rely exclusively on unimodal visual features for predicate classification. To address these challenges, we propose VOST-SGG, a VLM-aided one-stage ST-SGG framework that integrates the common sense reasoning capabilities of vision-language models (VLMs) into the ST-SGG pipeline. First, we introduce the dual-source query initialization strategy that disentangles what to attend to from where to attend, enabling semantically grounded what-where reasoning. Furthermore, we propose a multi-modal feature bank that fuses visual, textual, and spatial cues derived from VLMs for improved predicate classification. Extensive experiments on the Action Genome dataset demonstrate that our approach achieves state-of-the-art performance, validating the effectiveness of integrating VLM-aided semantic priors and multi-modal features for ST-SGG. We will release the code at https://github.com/LUNAProject22/VOST.

场景图生成视觉语言模型视频理解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。