将文本-视频检索的1对多关系拆解为多对一,提升匹配精度。
Text Proxy: Decomposing Retrieval from a 1-to-N Relationship into N 1-to-1 Relationships for Text-Video Retrieval
- 用多个文本代理替代单一查询,实现更精准的语义扩展。
- 在三个数据集上提升R@1指标2.0%至3.3%。
- 适合关注多模态对齐与检索精度的研究者。
文本-视频检索(TVR)近年来因预训练模型和大语言模型的应用取得显著进展。然而,由于视频与文本模态间的固有差异以及数据表示的不规则性,准确匹配仍具挑战。本文提出Text-Video-ProxyNet(TV-ProxyNet),将传统的1对多关系分解为N个独立的1对1关系。通过一系列文本代理取代单个文本查询,不仅扩展了查询范围,还提升了扩展精度。每个文本代理通过包含‘导演’与‘距离’控制机制的迭代过程生成,调节其相对于原查询的方向与距离。该设计增强了语义对齐能力,有效缓解多模态数据中的差异与噪声。在MSRVTT、DiDeMo和ActivityNet Captions三个基准上的实验表明,相比基线方法,TV-ProxyNet在R@1上提升2.0%至3.3%。其在MSRVTT和ActivityNet Captions上达到当前最优性能,在DiDeMo上提升2.0%,验证了该方法在增强语义映射与降低错误率方面的有效性。
原文摘要 · Abstract (English)
Text-video retrieval (TVR) has seen substantial advancements in recent years, fueled by the utilization of pre-trained models and large language models (LLMs). Despite these advancements, achieving accurate matching in TVR remains challenging due to inherent disparities between video and textual modalities and irregularities in data representation. In this paper, we propose Text-Video-ProxyNet (TV-ProxyNet), a novel framework designed to decompose the conventional 1-to-N relationship of TVR into N distinct 1-to-1 relationships. By replacing a single text query with a series of text proxies, TV-ProxyNet not only broadens the query scope but also achieves a more precise expansion. Each text proxy is crafted through a refined iterative process, controlled by mechanisms we term as the director and dash, which regulate the proxy's direction and distance relative to the original text query. This setup not only facilitates more precise semantic alignment but also effectively manages the disparities and noise inherent in multimodal data. Our experiments on three representative video-text retrieval benchmarks, MSRVTT, DiDeMo, and ActivityNet Captions, demonstrate the effectiveness of TV-ProxyNet. The results show an improvement of 2.0% to 3.3% in R@1 over the baseline. TV-ProxyNet achieved state-of-the-art performance on MSRVTT and ActivityNet Captions, and a 2.0% improvement on DiDeMo compared to existing methods, validating our approach's ability to enhance semantic mapping and reduce error propensity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。