通过动态调整文本视频对的语义增量,缓解对比学习中的优化张力。
Rebalancing Contrastive Alignment with Bottlenecked Semantic Increments in Text-Video Retrieval
- 引入可学习的跨模态增量Δ_ij,缓解文本与视频间的语义鸿沟导致的优化冲突。
- 在四个基准上提升检索准确率与鲁棒性,最佳结果达86.3% mAP。
- 适合关注跨模态对齐与噪声鲁棒性的研究者和工业应用开发者。
近期文本-视频检索进展主要依赖对比学习。然而,现有方法常忽略模态差距的影响,导致锚点表征出现原位优化(即优化张力),限制了对齐能力;同时,噪声难负样本进一步扭曲锚点语义。为此,我们提出GARE——一种感知模态间隙的检索框架,引入文本t_i与视频v_j之间的可学习、成对增量Δ_ij,重新分配梯度以缓解优化张力并吸收噪声。通过在信任域约束下对InfoNCE损失进行多变量一阶泰勒展开推导Δ_ij,证明其引导更新沿局部一致下降方向。一个基于语义差距的轻量级神经模块,跨批次耦合增量实现结构感知修正。此外,采用放宽压缩的变分信息瓶颈正则化Δ,增强稳定性与语义一致性。在四个基准上的实验表明,GARE持续提升对齐精度与鲁棒性,验证了感知间隙的张力缓解策略的有效性。代码已开源。
原文摘要 · Abstract (English)
Recent progress in text-video retrieval has been largely driven by contrastive learning. However, existing methods often overlook the effect of the modality gap, which causes anchor representations to undergo in-place optimization (i.e., optimization tension) that limits their alignment capacity. Moreover, noisy hard negatives further distort the semantics of anchors. To address these issues, we propose GARE, a Gap-Aware Retrieval framework that introduces a learnable, pair-specific increment $Δ_{ij}$ between text $t_i$ and video $v_j$, redistributing gradients to relieve optimization tension and absorb noise. We derive $Δ_{ij}$ via a multivariate first-order Taylor expansion of the InfoNCE loss under a trust-region constraint, showing that it guides updates along locally consistent descent directions. A lightweight neural module conditioned on the semantic gap couples increments across batches for structure-aware correction. Furthermore, we regularize $Δ$ through a variational information bottleneck with relaxed compression, enhancing stability and semantic consistency. Experiments on four benchmarks demonstrate that GARE consistently improves alignment accuracy and robustness, validating the effectiveness of gap-aware tension mitigation. Code is available at https://github.com/musicman217/GARE-text-video-retrieval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。