用大模型生成视觉描述，提升图文匹配效果
Visual Semantic Description Generation with MLLMs for Image-Text Matching
- 用多模态大模型生成图像语义描述，填补视觉与文本的鸿沟
- 在Flickr30K和MSCOCO上显著提升图文匹配准确率
- 零样本迁移能力强，适用于新闻、遥感等新领域
图像-文本匹配（ITM）旨在解决视觉与文本模态表示差异带来的对齐难题，前者为连续高维特征，后者为离散结构化文本。本文提出一种新框架，利用多模态大语言模型（MLLMs）作为视觉语义解析器，生成丰富的视觉语义描述（VSD），提供语义锚点以促进跨模态对齐。方法包含：(1) 实例级对齐，将视觉特征与VSD融合，增强图像表征的语言表达力；(2) 原型级对齐，通过VSD聚类确保类别级别一致性。该模块可无缝集成至现有ITM模型。在Flickr30K和MSCOCO上的大量实验表明性能显著提升，且在新闻和遥感等跨域任务中展现出出色的零样本泛化能力。代码与模型权重已开源。
原文摘要 · Abstract (English)
Image-text matching (ITM) aims to address the fundamental challenge of aligning visual and textual modalities, which inherently differ in their representations, continuous, high-dimensional image features vs. discrete, structured text. We propose a novel framework that bridges the modality gap by leveraging multimodal large language models (MLLMs) as visual semantic parsers. By generating rich Visual Semantic Descriptions (VSD), MLLMs provide semantic anchor that facilitate cross-modal alignment. Our approach combines: (1) Instance-level alignment by fusing visual features with VSD to enhance the linguistic expressiveness of image representations, and (2) Prototype-level alignment through VSD clustering to ensure category-level consistency. These modules can be seamlessly integrated into existing ITM models. Extensive experiments on Flickr30K and MSCOCO demonstrate substantial performance improvements. The approach also exhibits remarkable zero-shot generalization to cross-domain tasks, including news and remote sensing ITM. The code and model checkpoints are available at https://github.com/Image-Text-Matching/VSD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。