用Mamba模型融合时间信息，提升部分相关视频检索精度。
MamFusion: Multi-Mamba with Temporal Fusion for Partially Relevant Video Retrieval
- 多Mamba模块结合时序融合，捕捉长视频中的状态关联性。
- 在多个数据集上达到当前最优检索效果，显著优于基线方法。
- 适合需要精准定位部分相关视频的多媒体检索研究者。
部分相关视频检索（PRVR）是多媒体检索中的挑战性任务，旨在识别并召回与查询部分相关的未剪辑视频。本文针对长视频内容理解中的信息冗余问题，利用Mamba模块出色的长期状态空间建模能力和线性可扩展性，提出适用于PRVR任务的多Mamba时序融合框架（MamFusion）。该框架有效捕捉长视频中与状态相关的语义关联，并将其无缝融入文本-视频相关性理解，从而提升检索效果。具体地，引入了时序文本到视频融合（Temporal T-to-V Fusion）和视频到文本融合（Temporal V-to-T Fusion），显式建模文本查询与视频片段之间的时序关系，增强上下文感知能力与检索准确性。在大规模数据集上的大量实验表明，MamFusion在检索有效性方面达到当前最优水平。代码已开源：https://github.com/Vision-Multimodal-Lab-HZCU/MamFusion。
原文摘要 · Abstract (English)
Partially Relevant Video Retrieval (PRVR) is a challenging task in the domain of multimedia retrieval. It is designed to identify and retrieve untrimmed videos that are partially relevant to the provided query. In this work, we investigate long-sequence video content understanding to address information redundancy issues. Leveraging the outstanding long-term state space modeling capability and linear scalability of the Mamba module, we introduce a multi-Mamba module with temporal fusion framework (MamFusion) tailored for PRVR task. This framework effectively captures the state-relatedness in long-term video content and seamlessly integrates it into text-video relevance understanding, thereby enhancing the retrieval process. Specifically, we introduce Temporal T-to-V Fusion and Temporal V-to-T Fusion to explicitly model temporal relationships between text queries and video moments, improving contextual awareness and retrieval accuracy. Extensive experiments conducted on large-scale datasets demonstrate that MamFusion achieves state-of-the-art performance in retrieval effectiveness. Code is available at the link: https://github.com/Vision-Multimodal-Lab-HZCU/MamFusion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。