用傅里叶等方法替代注意力，让视频摘要更高效
EDSNet: Efficient-DSNet for Video Summarization
- 用傅里叶、小波和Nyströmformer替换传统注意力
- 在TVSum和SumMe上计算量显著降低，性能仍领先
- 适合资源受限场景下的实时视频摘要应用
当前视频摘要方法多依赖基于Transformer的架构，因其二次复杂度而消耗大量计算资源。本文通过改进直接摘要网络（DSNet），引入更高效的标记混合机制来解决这一问题。实验表明，用傅里叶变换、小波变换及Nyströmformer替代传统注意力机制，可显著提升效率与性能。同时，我们在区域建议网络中探索了多种池化策略，包括ROI池化、快速傅里叶变换池化与平铺池化。在TVSum和SumMe数据集上的实验结果表明，这些改进大幅降低了计算开销，同时保持了具有竞争力的摘要性能。本工作为视频摘要任务提供了更可扩展的解决方案。
原文摘要 · Abstract (English)
Current video summarization methods largely rely on transformer-based architectures, which, due to their quadratic complexity, require substantial computational resources. In this work, we address these inefficiencies by enhancing the Direct-to-Summarize Network (DSNet) with more resource-efficient token mixing mechanisms. We show that replacing traditional attention with alternatives like Fourier, Wavelet transforms, and Nyströmformer improves efficiency and performance. Furthermore, we explore various pooling strategies within the Regional Proposal Network, including ROI pooling, Fast Fourier Transform pooling, and flat pooling. Our experimental results on TVSum and SumMe datasets demonstrate that these modifications significantly reduce computational costs while maintaining competitive summarization performance. Thus, our work offers a more scalable solution for video summarization tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。