通过分析视频与文本信息密度差异,提升复杂视频查询的精准理解。
HUD: Hierarchical Uncertainty-Aware Disambiguation Network for Composed Video Retrieval
- 利用视频与文本信息密度差异,分层建模歧义消除机制。
- 在三个基准数据集上超越现有方法,显著提升检索准确率。
- 适用于视频和图像的复合查询任务,通用性强。
复合视频检索(CVR)是一项挑战性任务,需结合参考视频与修改文本进行目标视频检索。其核心在于理解多模态复合查询并实现精确的组合特征学习。通常视频模态包含比文本更丰富的语义信息,但现有工作忽略了两者间的信息密度差异,导致两大问题:修改主体指代歧义和细节语义关注不足,进而影响模型性能。为此,我们提出首个利用模态间信息密度差异的框架——分层不确定性感知消歧网络(HUD)。该框架包含三部分:(a) 整体代词消歧,(b) 原子级不确定性建模,(c) 整体到原子的对齐。通过整体跨模态交互捕捉重叠语义,并借助原子级跨模态交互实现细粒度对齐,有效完成对象消歧并强化细节语义聚焦,从而实现精准的组合特征学习。此外,该方法亦适用于复合图像检索(CIR),在三个基准数据集上均达到当前最优表现。代码已公开于 https://zivchen-ty.github.io/HUD.github.io/。
原文摘要 · Abstract (English)
Composed Video Retrieval (CVR) is a challenging video retrieval task that utilizes multi-modal queries, consisting of a reference video and modification text, to retrieve the desired target video. The core of this task lies in understanding the multi-modal composed query and achieving accurate composed feature learning. Within multi-modal queries, the video modality typically carries richer semantic content compared to the textual modality. However, previous works have largely overlooked the disparity in information density between these two modalities. This limitation can lead to two critical issues: 1) modification subject referring ambiguity and 2) limited detailed semantic focus, both of which degrade the performance of CVR models. To address the aforementioned issues, we propose a novel CVR framework, namely the Hierarchical Uncertainty-aware Disambiguation network (HUD). HUD is the first framework that leverages the disparity in information density between video and text to enhance multi-modal query understanding. It comprises three key components: (a) Holistic Pronoun Disambiguation, (b) Atomistic Uncertainty Modeling, and (c) Holistic-to-Atomistic Alignment. By exploiting overlapping semantics through holistic cross-modal interaction and fine-grained semantic alignment via atomistic-level cross-modal interaction, HUD enables effective object disambiguation and enhances the focus on detailed semantics, thereby achieving precise composed feature learning. Moreover, our proposed HUD is also applicable to the Composed Image Retrieval (CIR) task and achieves state-of-the-art performance across three benchmark datasets for both CVR and CIR tasks. The codes are available on https://zivchen-ty.github.io/HUD.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。