用语言引导图结构学习,让视频摘要更懂用户需求。
Language-Guided Graph Representation Learning for Video Summarization
- 构建前后向无向图,保留视频时序与语义关系。
- 引入双阈值图卷积,精准区分相关与无关帧。
- 支持文本指令生成摘要,推理快、模型小。
随着社交媒体视频内容激增,视频摘要成为多媒体处理的关键任务。现有方法难以捕捉视频的全局依赖关系,且缺乏多模态用户定制能力。此外,帧间时间邻近性不等于语义相似性。为此,我们提出语言引导图表示学习网络(LGRLN)。通过将视频帧转换为结构化图,保留时序顺序与上下文依赖,构建前向、后向及无向图以有效维持视频序列性和语义关联。设计了具有双阈值图卷积机制的图内关系推理模块,能区分节点间的语义相关性。同时,提出语言引导跨模态嵌入模块,根据文本描述生成摘要,并将输出建模为伯努利混合分布,使用EM算法求解。实验表明,该方法在多个基准上优于现有方法。此外,LGRLN将推理时间与模型参数分别减少87.8%和91.7%。代码与预训练模型已开源。
原文摘要 · Abstract (English)
With the rapid growth of video content on social media, video summarization has become a crucial task in multimedia processing. However, existing methods face challenges in capturing global dependencies in video content and accommodating multimodal user customization. Moreover, temporal proximity between video frames does not always correspond to semantic proximity. To tackle these challenges, we propose a novel Language-guided Graph Representation Learning Network (LGRLN) for video summarization. Specifically, we introduce a video graph generator that converts video frames into a structured graph to preserve temporal order and contextual dependencies. By constructing forward, backward and undirected graphs, the video graph generator effectively preserves the sequentiality and contextual relationships of video content. We designed an intra-graph relational reasoning module with a dual-threshold graph convolution mechanism, which distinguishes semantically relevant frames from irrelevant ones between nodes. Additionally, our proposed language-guided cross-modal embedding module generates video summaries with specific textual descriptions. We model the summary generation output as a mixture of Bernoulli distribution and solve it with the EM algorithm. Experimental results show that our method outperforms existing approaches across multiple benchmarks. Moreover, we proposed LGRLN reduces inference time and model parameters by 87.8% and 91.7%, respectively. Our codes and pre-trained models are available at https://github.com/liwrui/LGRLN.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。