arXiv:2503.10111cs.CV2025-03中稿 · SIGIR 2025被引 18

解决视频检索系统随时间更新时遗忘旧内容的问题。

Continual Text-to-Video Retrieval with Frame Fusion and Task-Aware Routing

  • 用帧融合适配器捕捉视频动态,保持模型灵活性。
  • 任务感知专家混合机制确保新旧查询与视频特征语义一致。
  • 适合需要持续更新的视频检索场景,如智能监控、内容推荐。

文本到视频检索(TVR)旨在根据文本查询检索相关视频。然而,随着视频内容不断演化,如何使TVR系统适应新数据仍是关键但未被充分探索的挑战。本文首次提出持续文本到视频检索(CTVR)基准,以应对现有方法的局限性。基于预训练模型(PTM)的TVR方法在适应新任务时难以保持模型可塑性,而现有持续学习(CL)方法则面临灾难性遗忘问题,导致历史查询与存储视频特征之间的语义错位。为此,我们提出FrameFusionMoE框架,包含两个核心组件:(1) 帧融合适配器(FFA),在保留模型可塑性的同时捕捉视频的时间动态;(2) 任务感知专家混合(TAME),确保跨任务查询与存储视频特征间的语义一致性。该框架实现了对新视频内容的有效适应,同时保留历史文本-视频相关性,缓解灾难性遗忘。我们在两个基准数据集上,在多种任务设置下全面评估了FrameFusionMoE,结果表明其优于现有CL和TVR方法,在处理连续视频流时仍能实现优异检索性能且对早期任务退化极小。

原文摘要 · Abstract (English)

Text-to-Video Retrieval (TVR) aims to retrieve relevant videos based on textual queries. However, as video content evolves continuously, adapting TVR systems to new data remains a critical yet under-explored challenge. In this paper, we introduce the first benchmark for Continual Text-to-Video Retrieval (CTVR) to address the limitations of existing approaches. Current Pre-Trained Model (PTM)-based TVR methods struggle with maintaining model plasticity when adapting to new tasks, while existing Continual Learning (CL) methods suffer from catastrophic forgetting, leading to semantic misalignment between historical queries and stored video features. To address these two challenges, we propose FrameFusionMoE, a novel CTVR framework that comprises two key components: (1) the Frame Fusion Adapter (FFA), which captures temporal video dynamics while preserving model plasticity, and (2) the Task-Aware Mixture-of-Experts (TAME), which ensures consistent semantic alignment between queries across tasks and the stored video features. Thus, FrameFusionMoE enables effective adaptation to new video content while preserving historical text-video relevance to mitigate catastrophic forgetting. We comprehensively evaluate FrameFusionMoE on two benchmark datasets under various task settings. Results demonstrate that FrameFusionMoE outperforms existing CL and TVR methods, achieving superior retrieval performance with minimal degradation on earlier tasks when handling continuous video streams. Our code is available at: https://github.com/JasonCodeMaker/CTVR.

视频检索持续学习多模态语义对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。