融合音视频信息，提升视频摘要生成效果
MF2Summ: Multimodal Fusion for Video Summarization with Temporal Alignment
- 通过跨模态注意力与时间对齐机制融合视觉和音频特征
- 在SumMe和TVSum数据集上分别提升F1分数1.9%和0.6%
- 适合需要多模态理解的视频摘要研究者使用
在线视频内容的快速增长催生了高效视频摘要技术的需求。传统方法多依赖单一模态（通常为视觉），难以完整捕捉视频语义。本文提出MF2Summ，一种基于多模态理解的视频摘要模型，整合视觉与音频信息。该模型采用五阶段流程：特征提取、跨模态注意力交互、特征融合、片段预测和关键帧选择。视觉特征由预训练的GoogLeNet提取，音频特征则通过SoundNet获取。核心融合机制包含跨模态Transformer与对齐引导的自注意力Transformer，用于建模模态间依赖关系与时间对应性。预测片段重要性、位置与中心度后，结合非极大值抑制（NMS）与核时间分割（KTS）算法进行关键帧选择。在SumMe与TVSum数据集上的实验表明，MF2Summ表现优异，相比DSNet模型的F1分数分别提升1.9%与0.6%，优于其他主流方法。
原文摘要 · Abstract (English)
The rapid proliferation of online video content necessitates effective video summarization techniques. Traditional methods, often relying on a single modality (typically visual), struggle to capture the full semantic richness of videos. This paper introduces MF2Summ, a novel video summarization model based on multimodal content understanding, integrating both visual and auditory information. MF2Summ employs a five-stage process: feature extraction, cross-modal attention interaction, feature fusion, segment prediction, and key shot selection. Visual features are extracted using a pre-trained GoogLeNet model, while auditory features are derived using SoundNet. The core of our fusion mechanism involves a cross-modal Transformer and an alignment-guided self-attention Transformer, designed to effectively model inter-modal dependencies and temporal correspondences. Segment importance, location, and center-ness are predicted, followed by key shot selection using Non-Maximum Suppression (NMS) and the Kernel Temporal Segmentation (KTS) algorithm. Experimental results on the SumMe and TVSum datasets demonstrate that MF2Summ achieves competitive performance, notably improving F1-scores by 1.9\% and 0.6\% respectively over the DSNet model, and performing favorably against other state-of-the-art methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。