解决文本视频检索中的模糊匹配问题,提升部分相关视频的召回效果。
Ambiguity-Restrained Text-Video Representation Learning for Partially Relevant Video Retrieval
- 通过不确定性和相似性检测模糊文本-视频对,构建多正例对比学习框架。
- 在文本-帧级别建模细粒度语义关系,显著提升部分相关视频检索准确率。
- 采用跨模型检测机制避免单模型误判,适合多媒体检索与视频理解研究者。
部分相关视频检索(PRVR)旨在找到与给定文本查询中特定片段相关的视频。传统训练假设每个文本查询仅对应一个视频,但本文指出文本与视频内容间存在固有模糊性,源于其概念范围差异。为此提出模糊约束表示学习(ARL)框架,首先基于不确定性与相似性标准识别模糊配对:不确定性反映实例是否包含数据集中普遍共享的上下文,相似性则衡量成对语义重叠程度。随后,利用检测出的模糊对,通过多层次多正例对比学习和双三元组边缘损失进行语义关系学习。进一步地,关注同一未剪辑视频内部的细粒度关系,突破传统文本-视频层级训练局限,实现文本-帧级别的增强学习。最后,引入跨模型模糊检测以缓解单一模型在检测模糊对时导致的误差传播问题。综合各项组件后,所提方法在PRVR任务上展现出显著有效性。
原文摘要 · Abstract (English)
Partially Relevant Video Retrieval~(PRVR) aims to retrieve a video where a specific segment is relevant to a given text query. Typical training processes of PRVR assume a one-to-one relationship where each text query is relevant to only one video. However, we point out the inherent ambiguity between text and video content based on their conceptual scope and propose a framework that incorporates this ambiguity into the model learning process. Specifically, we propose Ambiguity-Restrained representation Learning~(ARL) to address ambiguous text-video pairs. Initially, ARL detects ambiguous pairs based on two criteria: uncertainty and similarity. Uncertainty represents whether instances include commonly shared context across the dataset, while similarity indicates pair-wise semantic overlap. Then, with the detected ambiguous pairs, our ARL hierarchically learns the semantic relationship via multi-positive contrastive learning and dual triplet margin loss. Additionally, we delve into fine-grained relationships within the video instances. Unlike typical training at the text-video level, where pairwise information is provided, we address the inherent ambiguity within frames of the same untrimmed video, which often contains multiple contexts. This allows us to further enhance learning at the text-frame level. Lastly, we propose cross-model ambiguity detection to mitigate the error propagation that occurs when a single model is employed to detect ambiguous pairs for its training. With all components combined, our proposed method demonstrates its effectiveness in PRVR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。