让文本更懂视频帧间关系,提升跨模态匹配精度
EagleNet: Energy-Aware Fine-Grained Relationship Learning Network for Text-Video Retrieval
- 构建文本-帧图,学习细粒度语义关联
- 引入能量感知机制,精准捕捉真实图文对分布
- 用sigmoid损失替代softmax,训练更稳定
文本-视频检索任务得益于大规模视觉-语言预训练模型的发展。传统方法侧重视频表征或跨模态对齐,近期工作则转向增强文本表达以匹配视频的丰富语义。然而,这些方法仅关注文本与帧之间的交互,忽略了视频内部帧间的丰富关系,导致生成的扩展文本无法捕获帧上下文信息,造成文本与视频语义不一致。为此,我们提出能量感知细粒度关系学习网络(EagleNet),生成准确且上下文感知的文本嵌入。具体地,提出的细粒度关系学习机制(FRL)通过生成的文本候选与帧构建文本-帧图,学习文本与帧之间的关系,并据此聚合文本候选,形成融合帧上下文信息的增强文本嵌入。为进一步提升FRL中的细粒度关系学习,设计了能量感知匹配(EAM)来建模文本-帧交互的能量,从而准确捕捉真实图文对的分布。此外,为实现更有效的跨模态对齐和稳定训练,用sigmoid损失替代传统的基于softmax的对比损失。大量实验表明,EagleNet在MSRVTT、DiDeMo、MSVD和VATEX数据集上均表现出优越性能。代码已公开于https://github.com/draym28/EagleNet。
原文摘要 · Abstract (English)
Text-video retrieval tasks have seen significant improvements due to the recent development of large-scale vision-language pre-trained models. Traditional methods primarily focus on video representations or cross-modal alignment, while recent works shift toward enriching text expressiveness to better match the rich semantics in videos. However, these methods use only interactions between text and frames/video, and ignore rich interactions among the internal frames within a video, so the final expanded text cannot capture frame contextual information, leading to disparities between text and video. In response, we introduce Energy-Aware Fine-Grained Relationship Learning Network (EagleNet) to generate accurate and context-aware enriched text embeddings. Specifically, the proposed Fine-Grained Relationship Learning mechanism (FRL) first constructs a text-frame graph by the generated text candidates and frames, then learns relationships among texts and frames, which are finally used to aggregate text candidates into an enriched text embedding that incorporates frame contextual information. To further improve fine-grained relationship learning in FRL, we design Energy-Aware Matching (EAM) to model the energy of text-frame interactions and thus accurately capture the distribution of real text-video pairs. Moreover, for more effective cross-modal alignment and stable training, we replace the conventional softmax-based contrastive loss with the sigmoid loss. Extensive experiments have demonstrated the superiority of EagleNet across MSRVTT, DiDeMo, MSVD, and VATEX. Codes are available at https://github.com/draym28/EagleNet.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。