解决视觉语言模型中全局与局部表示的冲突,提升细粒度理解能力。
HarmoCLIP: Harmonizing Global and Regional Representations in Contrastive Vision-Language Models
- 引入显式细粒度语义监督,对齐文本片段与对应视觉区域。
- 在图像检索任务上提升69.78%,边界框分类准确率提高3.2%。
- 无需修改结构,可直接接入CLIP,适合需要精细感知的应用。
对比学习视觉语言预训练(CLIP)在多种视觉语言任务中表现出强大的泛化能力和优异性能。然而,由于缺乏区域级监督,CLIP在细粒度语义理解方面表现有限。尽管已有方法试图缓解此问题,却无意中破坏了全局对齐,导致局部感知提升的同时全局一致性下降,形成持续存在的权衡。本文提出HarmoCLIP,一种新型框架,旨在调和CLIP中的全局与区域表示。我们发现,局部文本与视觉语义间缺乏直接对齐是该权衡的根本原因。为此,HarmoCLIP引入显式细粒度语义监督项,直接对齐文本片段与其对应的视觉区域,有效连接图像区域空间与文本空间。为进一步强化局部表征能力,方法设计了一种新颖的区域-语言对齐监督策略,在不损害全局语义一致性的前提下促进细粒度语义学习。大量实验表明,HarmoCLIP在图像检索任务上达到最新水平(最高提升69.78%),在边界框分类任务上Top-1准确率提升3.2%,始终优于先前方法,提供了一种平衡、高效且即插即用的解决方案。代码已开源:https://github.com/Erosist/HarmoCLIP。
原文摘要 · Abstract (English)
Contrastive Language-Image Pre-training (CLIP) has demonstrated remarkable generalization ability and strong performance across a wide range of vision-language tasks. However, due to the lack of region-level supervision, CLIP exhibits limited fine-grained semantic understanding. Although several methods attempt to mitigate this issue, they unintentionally disrupt the global alignment, resulting in a persistent trade-off where improving local perception simultaneously degrades global coherence. In this paper, we propose HarmoCLIP, a novel framework designed to harmonize global and region representations within CLIP. We first identify that the absence of direct alignment between local textual and visual semantics is the fundamental cause of the trade-off. To address this, HarmoCLIP introduces an explicit fine-grained semantic supervision term that directly aligns textual segments with their corresponding visual regions, effectively bridging the image region space and the textual space. To further strengthen the representation capability at the local level, our method introduces a novel Region-Language Alignment supervision strategy that promotes fine-grained semantic learning without compromising global semantic consistency. Extensive experiments demonstrate that HarmoCLIP achieves state-of-the-art (improvement up to 69.78%) performance on the global task of retrieval and yields a substantial 3.2% improvement in Top-1 accuracy on the region task of bounding-box classification, consistently outperforming prior approaches while providing a balanced, efficient, and plug-and-play solution to the global-local trade-off in CLIP. Code is available at https://github.com/Erosist/HarmoCLIP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。