arXiv:2502.02885cs.CVcs.AI2025-02被引 7

用自学习生成更精准视频描述,提升图文检索准确率。

Expertized Caption Auto-Enhancement for Video-Text Retrieval

  • 通过自学习机制自动优化视频描述生成质量。
  • 在MSR-VTT等数据集上达到68.5%的最高检索准确率。
  • 适合需要高精度视频图文匹配的应用场景。

视频-文本检索因视频描述个性化不足导致信息错配,跨模态表示对齐困难,检索结果模糊。现有文本重写方法虽扩展表达形式,但语义丰富度仍不足。本文转而增强视觉表征,通过生成描述使视频表达更贴近文本,促进图文匹配。尽管多模态大模型具备强大内容转写能力,但需精心设计提示以确保生成描述的合理性与完整性。为此,本文提出一种自动描述增强方法,通过自学习提升表达质量,减少人为经验依赖;并设计专家化描述选择机制,为每段视频定制增强描述,充分挖掘描述增强潜力。方法完全数据驱动,避免繁重数据收集与计算负担,提升自适应性,摆脱词典依赖。在多个基准测试中表现优异,具体在MSR-VTT上取得68.5%的Top-1召回率,MSVD上达68.1%,DiDeMo上达62.0%。代码已公开于https://github.com/CaryXiang/ECA4VTR。

原文摘要 · Abstract (English)

Video-text retrieval has been stuck in the information mismatch caused by personalized and inadequate textual descriptions of videos. The substantial information gap between the two modalities hinders an effective cross-modal representation alignment, resulting in ambiguous retrieval results. Although text rewriting methods have been proposed to broaden text expressions, the modality gap remains significant, as the text representation space is hardly expanded with insufficient semantic enrichment.Instead, this paper turns to enhancing visual presentation, bridging video expression closer to textual representation via caption generation and thereby facilitating video-text matching.While multimodal large language models (mLLM) have shown a powerful capability to convert video content into text, carefully crafted prompts are essential to ensure the reasonableness and completeness of the generated captions. Therefore, this paper proposes an automatic caption enhancement method that improves expression quality and mitigates empiricism in augmented captions through self-learning.Additionally, an expertized caption selection mechanism is designed and introduced to customize augmented captions for each video, further exploring the utilization potential of caption augmentation.Our method is entirely data-driven, which not only dispenses with heavy data collection and computation workload but also improves self-adaptability by circumventing lexicon dependence and introducing personalized matching. The superiority of our method is validated by state-of-the-art results on various benchmarks, specifically achieving Top-1 recall accuracy of 68.5% on MSR-VTT, 68.1% on MSVD, and 62.0% on DiDeMo. Our code is publicly available at https://github.com/CaryXiang/ECA4VTR.

视频检索描述生成自学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。