arXiv:2506.20583cs.CVcs.AI2025-06被引 52

用图模型整合细粒度描述,让长视频事件更准确地生成一句话摘要。

Dense Video Captioning using Graph-based Sentence Summarization

  • 将事件片段拆解为短段,再通过图网络融合语义关系生成摘要
  • 在ActivityNet和YouCook II上均优于现有方法,尤其在长时序事件上提升显著
  • 适合需要精准理解复杂视频事件的场景,如自动剪辑、智能检索

近期,密集视频字幕生成在检测并描述长视频中所有事件方面取得了显著进展。尽管已有成果令人鼓舞,但大多数方法未能充分挖掘事件时间段内场景演变信息,导致在场景与物体随时间变化较长的提案中表现不佳。为此,我们提出一种基于图的分段-摘要(GPaS)框架,分两阶段进行:第一阶段将整个事件提案分割为短视频段,以更精细级别进行描述;第二阶段将各段生成的丰富描述句汇总为一句完整句子,概括整个事件。我们重点设计了“摘要”阶段,提出一种有效利用语义词间关系的框架。通过将语义词视为图节点,结合图卷积网络(GCN)与长短时记忆网络(LSTM),借助视觉线索学习其交互关系。提出了两种GCN-LSTM交互(GLI)模块,实现两者无缝融合。在ActivityNet Captions和YouCook II两个基准数据集上的大量实验验证了该方法的有效性,性能超越当前最先进方法。

原文摘要 · Abstract (English)

Recently, dense video captioning has made attractive progress in detecting and captioning all events in a long untrimmed video. Despite promising results were achieved, most existing methods do not sufficiently explore the scene evolution within an event temporal proposal for captioning, and therefore perform less satisfactorily when the scenes and objects change over a relatively long proposal. To address this problem, we propose a graph-based partition-and-summarization (GPaS) framework for dense video captioning within two stages. For the ``partition" stage, a whole event proposal is split into short video segments for captioning at a finer level. For the ``summarization" stage, the generated sentences carrying rich description information for each segment are summarized into one sentence to describe the whole event. We particularly focus on the ``summarization" stage, and propose a framework that effectively exploits the relationship between semantic words for summarization. We achieve this goal by treating semantic words as nodes in a graph and learning their interactions by coupling Graph Convolutional Network (GCN) and Long Short Term Memory (LSTM), with the aid of visual cues. Two schemes of GCN-LSTM Interaction (GLI) modules are proposed for seamless integration of GCN and LSTM. The effectiveness of our approach is demonstrated via an extensive comparison with the state-of-the-arts methods on the two benchmarks ActivityNet Captions dataset and YouCook II dataset.

视频字幕图神经网络事件摘要

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。