让文字和视频精准对齐,解决部分匹配难题。
T2VParser: Adaptive Decomposition Tokens for Partial Alignment in Text to Video Retrieval
- 用可学习的分解令牌,自适应提取多视角语义
- 在VideoText-5M数据集上,R@1达48.7%,提升显著
- 适合需要细粒度图文匹配的研究者
文本到视频检索旨在训练模型精确对齐视觉内容与文本描述。尽管图像-文本预训练模型(如CLIP)展现出强大的多模态知识,现有工作主要将其知识扩展至视频-文本任务。然而,视频通常包含比图像更丰富的信息,而当前视频-文本数据集中,文本描述仅能反映视频内容的一部分,导致视频-文本匹配存在部分错位。因此,直接对齐文本与视频的完整表示会引入错误监督,忽略模态间信息量的不等价性。本文提出T2VParser,从文本和视频中提取多视角语义表示,实现自适应的语义对齐而非整体对齐。为跨模态提取对应表示,引入共享的可学习分解令牌。T2VParser的目标是在保留预训练模型知识的同时,强调文本与视频间的精准对齐。实验表明,通过有效的跨模态内容分解,T2VParser实现了准确的部分对齐。代码已开源。
原文摘要 · Abstract (English)
Text-to-video retrieval essentially aims to train models to align visual content with textual descriptions accurately. Due to the impressive general multimodal knowledge demonstrated by image-text pretrained models such as CLIP, existing work has primarily focused on extending CLIP knowledge for video-text tasks. However, videos typically contain richer information than images. In current video-text datasets, textual descriptions can only reflect a portion of the video content, leading to partial misalignment in video-text matching. Therefore, directly aligning text representations with video representations can result in incorrect supervision, ignoring the inequivalence of information. In this work, we propose T2VParser to extract multiview semantic representations from text and video, achieving adaptive semantic alignment rather than aligning the entire representation. To extract corresponding representations from different modalities, we introduce Adaptive Decomposition Tokens, which consist of a set of learnable tokens shared across modalities. The goal of T2VParser is to emphasize precise alignment between text and video while retaining the knowledge of pretrained models. Experimental results demonstrate that T2VParser achieves accurate partial alignment through effective cross-modal content decomposition. The code is available at https://github.com/Lilidamowang/T2VParser.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。