提出不均等事件建模框架，提升文本查询视频检索精度
Uneven Event Modeling for Partially Relevant Video Retrieval
- 基于时序依赖与语义相似性动态分割视频事件
- 通过上下文感知模块增强事件表征，对齐文本关键帧
- 适用于需要精准定位相关片段的视频检索任务
给定文本查询，部分相关视频检索（PRVR）旨在找出包含相关片段的未剪辑视频，其中事件建模对划分视频为对应文本的时序事件至关重要。以往方法通常将视频切分为固定数量的等长片段，导致事件边界模糊；且依赖平均池化计算事件表示，不可避免引入错位。为此，我们提出不均等事件建模（UEM）框架用于PRVR。首先引入渐进式分组视频分割（PGVS）模块，结合相邻帧间的时序依赖与语义相似性，迭代生成清晰的事件边界。此外，提出上下文感知事件精炼（CAER）模块，基于文本交叉注意力条件化事件表示，使事件表征聚焦于最相关的帧，实现更精确的文本-视频对齐。大量实验表明，该方法在两个PRVR基准上达到当前最优性能。代码已公开于https://github.com/Sasa77777779/UEM.git。
原文摘要 · Abstract (English)
Given a text query, partially relevant video retrieval (PRVR) aims to retrieve untrimmed videos containing relevant moments, wherein event modeling is crucial for partitioning the video into smaller temporal events that partially correspond to the text. Previous methods typically segment videos into a fixed number of equal-length clips, resulting in ambiguous event boundaries. Additionally, they rely on mean pooling to compute event representations, inevitably introducing undesired misalignment. To address these, we propose an Uneven Event Modeling (UEM) framework for PRVR. We first introduce the Progressive-Grouped Video Segmentation (PGVS) module, to iteratively formulate events in light of both temporal dependencies and semantic similarity between consecutive frames, enabling clear event boundaries. Furthermore, we also propose the Context-Aware Event Refinement (CAER) module to refine the event representation conditioned the text's cross-attention. This enables event representations to focus on the most relevant frames for a given text, facilitating more precise text-video alignment. Extensive experiments demonstrate that our method achieves state-of-the-art performance on two PRVR benchmarks. Code is available at https://github.com/Sasa77777779/UEM.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。