arXiv:2509.05604cs.CVcs.AI2025-09IJCV被引 1

用语言引导的图网络,让视频摘要更懂物体和语义关系。

Language-guided Recursive Spatiotemporal Graph Modeling for Video Summarization

  • 把帧和物体建模为时空图节点,用语言查询增强语义关系
  • 递归优化图结构,在多个数据集上达到最优性能
  • 适合需要理解复杂场景语义的视频摘要任务

视频摘要旨在选取视觉多样且能代表全片内容的关键帧。以往方法多关注帧间全局时序关联,但细粒度视觉实体(如物体)同样对视频主内容至关重要。近年来兴起的语言引导视频摘要需深入理解复杂现实场景。本文将视频摘要视为语言引导的时空图建模问题,提出递归时空图网络 VideoGraph,将物体与帧分别作为空间和时间图的节点。各图内节点通过边连接并聚合,表示其语义关联。为避免边仅由视觉相似性决定,引入从视频中提取的语言查询,增强节点表示的语义信息。同时采用递归策略不断优化初始图结构,准确判定每帧是否为关键帧。实验表明,VideoGraph在通用与查询聚焦视频摘要任务中,无论监督还是无监督设置下均达当前最佳表现。代码已公开于 https://github.com/park-jungin/videograph。

原文摘要 · Abstract (English)

Video summarization aims to select keyframes that are visually diverse and can represent the whole story of a given video. Previous approaches have focused on global interlinkability between frames in a video by temporal modeling. However, fine-grained visual entities, such as objects, are also highly related to the main content of the video. Moreover, language-guided video summarization, which has recently been studied, requires a comprehensive linguistic understanding of complex real-world videos. To consider how all the objects are semantically related to each other, this paper regards video summarization as a language-guided spatiotemporal graph modeling problem. We present recursive spatiotemporal graph networks, called VideoGraph, which formulate the objects and frames as nodes of the spatial and temporal graphs, respectively. The nodes in each graph are connected and aggregated with graph edges, representing the semantic relationships between the nodes. To prevent the edges from being configured with visual similarity, we incorporate language queries derived from the video into the graph node representations, enabling them to contain semantic knowledge. In addition, we adopt a recursive strategy to refine initial graphs and correctly classify each frame node as a keyframe. In our experiments, VideoGraph achieves state-of-the-art performance on several benchmarks for generic and query-focused video summarization in both supervised and unsupervised manners. The code is available at https://github.com/park-jungin/videograph.

视频摘要图神经网络语言引导时空建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。