arXiv:2605.24470cs.CV2026-05被引 7

针对第一人称视频检索,引入时序建模与两阶段重排序提升跨模态匹配精度。

TempRet: Temporal Enhancement and Two-Stage Reranking for CVPR 2026 EPIC-KITCHENS-100 Multi-Instance Retrieval Challenge

论文配图:TempRet: Temporal Enhancement and Two-Stage Reranking for CVPR 2026 EPIC-KITCHENS-100 Multi-Instance Retrieval Challenge
图 1 · 摘自论文原文
  • 用时序变换器建模视频帧间动态关系,增强时间语义理解。
  • 两阶段重排序先粗筛再精调,结合软标签提升匹配精度。
  • 适合关注第一人称视频与跨模态检索的研究者参考。

视频-文本检索在大规模视觉语言预训练推动下取得显著进展,但现有方法大多沿袭图像-文本检索的隐含假设:视觉语义可逐帧捕捉。这一假设忽略了第一人称视频的时间动态性。EPIC-KITCHENS-100 多实例检索(MIR)挑战进一步提高标准,提供软标签相关矩阵而非二值标签,要求模型能解析跨模态的分级语义对应关系。本文提出名为 TempRet 的解决方案,基于 CLIP 双编码器骨干网络,引入两个关键组件:首先,仅作用于视频侧的时序变换器,通过可学习位置编码和多头自注意力机制建模帧级 CLIP 特征间的依赖关系;其次,两阶段重排序流程:先通过双编码器获取前 K 名候选,再使用带图像-文本匹配(ITM)头的交叉编码器对得分进行精修。整个系统采用对称多相似性损失训练,以利用挑战提供的软标签相关矩阵。在 EK-100 MIR 基准上,该方法实现 67.97% 平均 mAP 与 82.92% 平均 nDCG,验证了时序建模与跨模态精修对第一人称视频检索的有效性。

原文摘要 · Abstract (English)

Video-text retrieval has witnessed remarkable progress driven by large-scale vision-language pretraining, yet most existing approaches inherit an implicit assumption from image-text retrieval: that visual semantics can be captured frame-by-frame. This assumption overlooks the temporal dynamics of egocentric videos. The EPIC-KITCHENS-100 Multi-Instance Retrieval (MIR) challenge further raises the bar by providing soft-label relevance matrices rather than binary labels, demanding models that can resolve graded semantic correspondences across modalities. In this report, we present our solution, termed TempRet, to the CVPR 2026 EPIC-KITCHENS-100 MIR challenge. Our approach builds upon a CLIP-based dual-encoder backbone and introduces two key components to address the temporal and cross-modal challenges. First, a temporal transformer operates exclusively on the video side, modeling inter-frame dependencies through learnable positional encodings and multi-head self-attention over frame-level CLIP features. Second, a two-stage reranking pipeline first retrieves Top-K candidates via the dual-encoder, then refines their scores using a cross-encoder equipped with an Image-Text Matching (ITM) head. The entire system is trained with Symmetric Multi-Similarity Loss to exploit the soft-label relevance matrices provided by the challenge. Our method achieves 67.97% average mAP and 82.92% average nDCG on the EK-100 MIR benchmark, demonstrating the effectiveness of temporal modeling and cross-modal refinement for egocentric video retrieval.

视频检索时序建模跨模态匹配

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。