用视觉提示辅助摘要生成,让长视频自动生成精准描述。
Show, Tell and Summarize: Dense Video Captioning Using Visual Cue Aided Sentence Summarization
- 将视频分段后生成句子,再通过双阶段LSTM融合视觉特征摘要
- 在ActivityNet数据集上显著提升稠密视频字幕精度
- 适合需要自动视频理解与摘要的场景
本文提出一种分而摘要(DaS)框架用于稠密视频字幕生成。先将无剪辑长视频划分为多个事件提案,每个提案包含若干短片段;对每个片段提取视觉特征(如C3D特征),并使用现有图像/视频字幕方法生成一句描述。由于生成句包含丰富语义信息,我们把稠密视频字幕任务建模为视觉提示辅助的句子摘要问题,提出一种新型两阶段LSTM模型,配备层次化注意力机制,结合所有片段的视觉特征,将生成句汇总为一句完整描述。第一阶段LSTM作为编码器,融合所有句子的语义词和片段视觉特征,有效总结事件信息;第二阶段作为解码器,基于第一阶段输出和全部视觉特征生成最终描述句。在ActivityNet Captions数据集上的大量实验验证了该框架的有效性。
原文摘要 · Abstract (English)
In this work, we propose a division-and-summarization (DaS) framework for dense video captioning. After partitioning each untrimmed long video as multiple event proposals, where each event proposal consists of a set of short video segments, we extract visual feature (e.g., C3D feature) from each segment and use the existing image/video captioning approach to generate one sentence description for this segment. Considering that the generated sentences contain rich semantic descriptions about the whole event proposal, we formulate the dense video captioning task as a visual cue aided sentence summarization problem and propose a new two stage Long Short Term Memory (LSTM) approach equipped with a new hierarchical attention mechanism to summarize all generated sentences as one descriptive sentence with the aid of visual features. Specifically, the first-stage LSTM network takes all semantic words from the generated sentences and the visual features from all segments within one event proposal as the input, and acts as the encoder to effectively summarize both semantic and visual information related to this event proposal. The second-stage LSTM network takes the output from the first-stage LSTM network and the visual features from all video segments within one event proposal as the input, and acts as the decoder to generate one descriptive sentence for this event proposal. Our comprehensive experiments on the ActivityNet Captions dataset demonstrate the effectiveness of our newly proposed DaS framework for dense video captioning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。