通过多层视觉特征挖掘，提升视频文本检索精度
Delving Deeper: Hierarchical Visual Perception for Robust Video-Text Retrieval
- 从视觉编码器中间层提取并优化特征，逐级提炼关键视觉概念
- 在MSRVTT、DiDeMo、ActivityNet上达到新最优性能
- 适合关注视频语义理解与跨模态对齐的研究者
视频-文本检索（VTR）旨在用自然语言查询定位相关视频。现有方法多基于CLIP等预训练模型，受限于视频固有的冗余性及对粗粒度最终层特征的依赖，匹配精度不足。为此，我们提出HVP-Net（层级视觉感知网络），通过从视觉编码器多个中间层提取并精炼特征，逐步从原始图像块令牌中提炼出不同语义层级的显著视觉概念，有效缓解冗余同时保留关键对齐细节。该方法构建了更鲁棒的视频表示，在MSRVTT、DiDeMo和ActivityNet等挑战性基准上取得新最优结果。研究验证了利用层级特征对提升视频-文本检索的有效性。代码已开源：https://github.com/boyun-zhang/HVP-Net。
原文摘要 · Abstract (English)
Video-text retrieval (VTR) aims to locate relevant videos using natural language queries. Current methods, often based on pre-trained models like CLIP, are hindered by video's inherent redundancy and their reliance on coarse, final-layer features, limiting matching accuracy. To address this, we introduce the HVP-Net (Hierarchical Visual Perception Network), a framework that mines richer video semantics by extracting and refining features from multiple intermediate layers of a vision encoder. Our approach progressively distills salient visual concepts from raw patch-tokens at different semantic levels, mitigating redundancy while preserving crucial details for alignment. This results in a more robust video representation, leading to new state-of-the-art performance on challenging benchmarks including MSRVTT, DiDeMo, and ActivityNet. Our work validates the effectiveness of exploiting hierarchical features for advancing video-text retrieval. Our codes are available at https://github.com/boyun-zhang/HVP-Net.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。