用原型桥接文本与视频,高效对齐多层级语义,提升检索准确率。
PHA-Net: Prototype-based Hierarchical Alignment Network for Text-Video Retrieval

- 引入共享原型作为跨模态桥梁,统一优化文本与视频表征。
- 在四个数据集上平均召回率提升4.9%至19.2%,最高达19.2%。
- 适合关注视频检索、跨模态对齐的开发者与研究者。
随着大规模图文预训练模型(如CLIP)的出现,文本-视频检索近年取得显著进展。现有最佳方法同时在个体、局部和全局层面进行跨模态语义对齐,但面临简洁文本与丰富视频之间的内在语义错配问题。传统方法通过集成多层语言-视频注意力模块实现层级对齐,但计算成本过高。本文提出原型驱动的分层对齐网络(PHA-Net),通过多个模态共享原型高效优化文本与视频表示。针对聚类令牌中语义分布不均的问题,设计原型引导的令牌融合模块,强化强语义令牌、抑制弱语义令牌。此外,引入原型对比损失,促使同一原型下的文本与视觉原型间更相似,不同原型间差异更大。在四个基准数据集上的实验表明,PHA-Net在MSR-VTT(8.8%)、ActivityNet(19.2%)、VATEX(0.7%)和Charades(4.9%)上均实现显著提升,代码已开源。
原文摘要 · Abstract (English)
With the emergence of large-scale image-text pre-training models, e.g., CLIP, text-video retrieval has experienced substantial advances in recent years. Existing best-performing methods involve aligning cross-modal semantics at individual, local, and global levels simultaneously, raising concerns about the intrinsic semantic mismatch between concise texts and rich videos. A canonical approach is to integrate multiple language-video attention modules into the hierarchical framework while this paradigm only optimizes visual representations with prohibitive computational costs. In this paper, we propose a new prototype-based hierarchical alignment network (PHA-Net) to align individual/local/global level representations across modalities. Concretely, we introduce multiple modality-shared prototypes as the bridge to efficiently optimize text and video representations for cross-modal alignment. Then, we argue that the imbalanced semantic distribution in clustered tokens may undermine retrieval performance, as tokens with weak semantics are of little interest. To reduce the impact of these tokens, a proposed prototype-supported token merge module is responsible for enhancing tokens with strong semantics and suppressing others with weak semantics via prototype semantics guidance. Moreover, we devise a prototype contrastive loss to encourage textual and visual prototypes to focus on different semantic information. The idea of this auxiliary loss is to ensure higher similarity between textual and visual prototypes from the same prototype than those from different prototypes. Extensive experiments on four benchmarks confirm the effectiveness of our PHA-Net, which achieves significant improvements in the sum of all recalls on MSR-VTT (8.8%), ActivityNet (19.2%), VATEX (0.7%), and Charades (4.9%). Code is available at https://github.com/JingXiaolun/PHA-Net.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。