arXiv:2501.00882cs.CV2025-01被引 8

提出全注意力视频摘要模型,兼顾长程依赖与计算效率。

FullTransNet: Full Transformer with Local-Global Attention for Video Summarization

  • 采用编码器-解码器结构的完整Transformer,提升序列建模能力。
  • 引入局部-全局稀疏注意力,计算量降低但长程依赖捕捉更优。
  • 在SumMe和TVSum上分别达54.4%和63.9%的F-score,性能领先。

视频摘要旨在生成紧凑、信息丰富且具代表性的原始视频概要,对视频浏览、分析与理解至关重要。主流方法多依赖循环或卷积神经网络,近年也采用仅编码器的Transformer架构。然而这些方法在并行性、长程依赖建模及显式生成能力方面存在局限。为此,我们提出名为FullTransNet的类Transformer架构,包含双重设计:首先,采用编码器-解码器结构的完整Transformer,其专为序列转换任务设计,直接应用于视频摘要既直观又高效;其次,用局部-全局稀疏注意力替代标准全注意力机制,使模型在捕捉长程依赖的同时显著降低计算开销。该机制仅作用于计算量最大的编码器端,进一步提升效率。在SumMe和TVSum两个常用基准数据集上的实验表明,模型分别取得54.4%和63.9%的F-score,且计算与内存消耗较低。结果优于次优方法0.1%和0.3%,验证了FullTransNet的有效性与高效性。

原文摘要 · Abstract (English)

Video summarization aims to generate a compact, informative, and representative synopsis of raw videos, which is crucial for browsing, analyzing, and understanding video content. Dominant approaches in video summarization primarily rely on recurrent or convolutional neural networks, and more recently on encoder-only transformer architectures. However, these methods typically suffer from several limitations in parallelism, modeling long-range dependencies, and providing explicit generative capabilities. To address these issues, we propose a transformer-like architecture named FullTransNet with two-fold ideas. First, it uses a full transformer with an encoder-decoder structure as an alternative architecture for video summarization. As the full transformer is specifically designed for sequence transduction tasks, its direct application to video summarization is both intuitive and effective. Second, it replaces the standard full attention mechanism with a combination of local and global sparse attention, enabling the model to capture long-range dependencies while significantly reducing computational costs. This local-global sparse attention is applied exclusively at the encoder side, where the majority of computations occur, further enhancing efficiency. Extensive experiments on two widely used benchmark datasets, SumMe and TVSum, demonstrate that our model achieves F-scores of 54.4% and 63.9%, respectively, while maintaining relatively low computational and memory requirements. These results surpass the second-best performing methods by 0.1% and 0.3%, respectively, verifying the effectiveness and efficiency of FullTransNet.

视频摘要Transformer稀疏注意力

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。