通过文本引导的多粒度对比学习,提升图文检索精度。
TC-MGC: Text-Conditioned Multi-Grained Contrastive Learning for Text-Video Retrieval
- 用语言注意力生成文本相关的视频表征,避免无关语义干扰。
- 在多个数据集上超越X-CLIP模型,最高提升2.8%(绝对1.3%)。
- 适合追求高精度图文检索的开发者和研究者。
受粗粒度或细粒度对比学习成功的启发,多粒度对比学习方法逐渐兴起,致力于整合不同粒度的对比信息。然而,由于视频语义范围更广,非文本相关的视频表征可能包含未被文本描述的信息,从而阻碍模型捕捉精确的跨模态语义对应关系。为此,我们提出一种文本条件化的多粒度对比学习框架——TC-MGC。具体地,模型采用语言-视频注意力模块,根据词语和文本对帧的注意力权重生成聚合帧与视频表征。为减少不必要的相似性交互并降低交互相似性聚合(ISA)模块的可训练参数,设计了相似性重构(SR)模块,以识别关键相似性并重组跨模态相似性向量与矩阵。进一步指出多粒度相似性间的不平衡问题可能导致过表达与欠表达,因此引入辅助的相似性去相关正则化(SDR)损失,通过最小化匹配文本-视频对的相似性方差来促进协同关系利用。最后,提出线性Softmax聚合(LSA)模块,显式强化多组相似性之间的交互,推动多粒度信息的有效使用。实验表明,TC-MGC在多个图文检索基准上表现优异,在MSR-VTT、DiDeMo和VATEX上的文本到视频检索R@1分别优于X-CLIP模型+2.8%(绝对1.3%)、+2.2%(绝对1.0%)、+1.5%(绝对0.9%)。代码已公开于https://github.com/JingXiaolun/TC-MGC。
原文摘要 · Abstract (English)
Motivated by the success of coarse-grained or fine-grained contrast in text-video retrieval, there emerge multi-grained contrastive learning methods which focus on the integration of contrasts with different granularity. However, due to the wider semantic range of videos, the text-agnostic video representations might encode misleading information not described in texts, thus impeding the model from capturing precise cross-modal semantic correspondence. To this end, we propose a Text-Conditioned Multi-Grained Contrast framework, dubbed TC-MGC. Specifically, our model employs a language-video attention block to generate aggregated frame and video representations conditioned on the word's and text's attention weights over frames. To filter unnecessary similarity interactions and decrease trainable parameters in the Interactive Similarity Aggregation (ISA) module, we design a Similarity Reorganization (SR) module to identify attentive similarities and reorganize cross-modal similarity vectors and matrices. Next, we argue that the imbalance problem among multigrained similarities may result in over- and under-representation issues. We thereby introduce an auxiliary Similarity Decorrelation Regularization (SDR) loss to facilitate cooperative relationship utilization by similarity variance minimization on matching text-video pairs. Finally, we present a Linear Softmax Aggregation (LSA) module to explicitly encourage the interactions between multiple similarities and promote the usage of multi-grained information. Empirically, TC-MGC achieves competitive results on multiple text-video retrieval benchmarks, outperforming X-CLIP model by +2.8% (+1.3%), +2.2% (+1.0%), +1.5% (+0.9%) relative (absolute) improvements in text-to-video retrieval R@1 on MSR-VTT, DiDeMo and VATEX, respectively. Our code is publicly available at https://github.com/JingXiaolun/TC-MGC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。