通过语义增强的补丁精简框架,提升视觉与文本的细粒度对齐效果。
SEPS: Semantic-enhanced Patch Slimming Framework for fine-grained cross-modal alignment
- 分两阶段融合稠密与稀疏文本语义,识别关键视觉补丁。
- 在Flickr30K和MS-COCO上比现有方法提升23%-86%的rSum指标。
- 适合需要精准图文匹配的视觉问答与跨模态检索场景。
细粒度跨模态对齐旨在建立视觉与语言之间的精确局部对应关系,是视觉问答及相关多模态应用的基础。现有方法面临补丁冗余与歧义问题,源于模态间固有的信息密度差异。近期,多模态大语言模型(MLLMs)凭借强大的语义生成能力成为潜在解决方案,但其密集文本输出可能与原始稀疏描述产生冲突。此外,准确量化丰富视觉补丁与简洁文本描述之间的语义相关性仍是核心挑战。为此,本文提出语义增强的补丁精简(SEPS)框架,系统性解决补丁冗余与歧义问题。该方法采用两阶段机制,整合稠密与稀疏文本的统一语义,实现关键视觉补丁的识别;同时利用感知相关性的选择策略与均值计算,突出重要补丁-词语对应关系,从而提升跨模态相似性评估性能。在Flickr30K和MS-COCO数据集上的综合实验表明,SEPS在多种模型架构下,于rSum指标上超越现有方法23%-86%,尤其在文本到图像检索场景中表现显著。代码已开源:https://github.com/Sweet4tars/seps.git。
原文摘要 · Abstract (English)
Fine-grained cross-modal alignment aims to establish precise local correspondences between vision and language, forming a cornerstone for visual question answering and related multimodal applications. Current approaches face challenges in addressing patch redundancy and ambiguity, which arise from the inherent information density disparities across modalities. Recently, Multimodal Large Language Models (MLLMs) have emerged as promising solutions to bridge this gap through their robust semantic generation capabilities. However, the dense textual outputs from MLLMs may introduce conflicts with the original sparse captions. Furthermore, accurately quantifying semantic relevance between rich visual patches and concise textual descriptions remains a core challenge. To overcome these limitations, we introduce the Semantic-Enhanced Patch Slimming (SEPS) framework, which systematically addresses patch redundancy and ambiguity. Our approach employs a two-stage mechanism to integrate unified semantics from both dense and sparse texts, enabling the identification of salient visual patches. Additionally, it leverages relevance-aware selection with mean value computation to highlight crucial patch-word correspondences, thereby improving cross-modal similarity assessment. Comprehensive experiments on Flickr30K and MS-COCO datasets validate that SEPS achieves superior performance, surpassing existing approaches by 23\%-86\% in rSum across diverse model architectures, with notable enhancements in text-to-image retrieval scenarios. Our implementation is available at https://github.com/Sweet4tars/seps.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。