arXiv:2609.02204cs.CV2026-09被引 2

让视频图文检索更懂时间:用专家分工和时序信息增强模型理解

TAME: Temporal-Aware Mixture-of-Experts for Text-Video Retrieval

论文配图:TAME: Temporal-Aware Mixture-of-Experts for Text-Video Retrieval
图 1 · 摘自论文原文
  • 引入稀疏专家网络,让不同视觉专家负责不同帧特征,保持语义对齐
  • 通过帧间时序令牌,捕捉长距离时间依赖,同时保留局部细节
  • 设计分阶段时序融合模块,提升句子与视频的匹配精度,适合视频理解任务

文本-视频检索(TVR)旨在根据自然语言查询找到匹配的视频,但将图像-文本模型如CLIP扩展到视频时,受限于缺乏时序建模能力。视频在外观和运动上存在帧级差异,将所有帧压缩为单一表示会掩盖时序结构与语义变化。为此,我们提出基于CLIP的时序感知混合专家框架TAME,联合建模帧级结构与时间关系。首先,在双编码器中集成稀疏混合专家(MoE)层,并在视觉分支采用帧一致路由,使专家按帧级视觉模式专业化,同时保持原有跨模态对齐。其次,引入帧-时序(FT)令牌,聚合全局跨帧信息并反馈至每帧,使视觉编码器在不损害局部细节的前提下捕获长程时序依赖。第三,设计跨时序交互与聚合(CTIA)模块,通过分阶段时序过滤与融合,优化帧级句-视频相似度。在标准TVR基准测试中,TAME持续优于基于CLIP的基线模型。在MSR-VTT上,相较CLIP4Clip提升R@1达4.0;在DiDeMo、MSVD、LSMDC和ActivityNet上也取得稳定增益。代码已开源。

原文摘要 · Abstract (English)

Text-Video Retrieval (TVR) retrieves videos that match a natural-language query, but extending image-text models such as CLIP to videos is fundamentally limited by the lack of temporal modeling. Videos exhibit frame-wise heterogeneity in appearance and motion, and compressing all frames into a single representation often obscures temporal structure and semantic transitions. To address this, we propose Temporal-Aware Mixture-of-Experts for Text-Video Retrieval (TAME), a CLIP-based framework that jointly models frame-level structure and temporal relations. First, we integrate sparse Mixture-of-Experts (MoE) layers into both CLIP encoders and apply frame-consistent routing on the vision branch so that experts specialize according to frame-level visual patterns while preserving the original vision-language alignment. Second, we introduce Frame-Temporal (FT) tokens that aggregate global cross-frame information and feed it back to each frame, enabling the visual encoder to capture long-range temporal dependencies without harming local details. Third, we design a Cross-Temporal Interaction and Aggregation (CTIA) module that refines frame-wise sentence-video similarities through staged temporal filtering and fusion. Experiments on standard TVR benchmarks show that TAME consistently improves over CLIP-based baselines. On MSR-VTT, it improves R@1 by 4.0 over CLIP4Clip, and also achieves consistent gains on DiDeMo, MSVD, LSMDC, and ActivityNet. The code is available at https://github.com/sejong-rcv/TAME.

视频检索时序建模混合专家CLIP

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。