用提示金字塔提升视频检索精度，让模型理解片段级相关性。
ProPy: Building Interactive Prompt Pyramids upon CLIP for Partially Relevant Video Retrieval
- 构建分层提示金字塔，捕捉多粒度事件语义
- 通过祖先-后代交互机制实现动态语义关联，性能超越已有方法
- 适用于需要精准定位视频片段的检索任务
部分相关视频检索（PRVR）是一项实用但具有挑战性的任务，需根据仅与视频特定片段相关的查询检索视频。现有方法多基于单模态特征建模，而强大的预训练视觉语言模型如CLIP在此领域尚未被充分探索。为此，我们提出ProPy，一种针对PRVR系统性适配CLIP架构的新模型。受多粒度事件语义相关性启发，ProPy引入两项关键创新：(1) 提示金字塔结构，用于组织事件提示以捕获多粒度语义；(2) 基于该金字塔的祖先-后代交互机制，实现事件间的动态语义交互。这些设计使ProPy在三个公开数据集上达到当前最优性能，显著优于先前模型。代码已开源：https://github.com/BUAAPY/ProPy。
原文摘要 · Abstract (English)
Partially Relevant Video Retrieval (PRVR) is a practical yet challenging task that involves retrieving videos based on queries relevant to only specific segments. While existing works follow the paradigm of developing models to process unimodal features, powerful pretrained vision-language models like CLIP remain underexplored in this field. To bridge this gap, we propose ProPy, a model with systematic architectural adaption of CLIP specifically designed for PRVR. Drawing insights from the semantic relevance of multi-granularity events, ProPy introduces two key innovations: (1) A Prompt Pyramid structure that organizes event prompts to capture semantics at multiple granularity levels, and (2) An Ancestor-Descendant Interaction Mechanism built on the pyramid that enables dynamic semantic interaction among events. With these designs, ProPy achieves SOTA performance on three public datasets, outperforming previous models by significant margins. Code is available at https://github.com/BUAAPY/ProPy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。