让CLIP更好理解长文本描述，提升图文细粒度对齐效果
MulCLIP: A Multi-level Alignment Framework for Enhancing Fine-grained Long-context CLIP
- 分层级对齐：全局+局部结合，增强长文本与图像区域的对应关系
- 在多个基准上显著提升下游任务性能，优于依赖区域建议的方法
- 无需额外区域标注，适合实际场景中复杂长描述的图文理解
视觉语言模型如CLIP在短句描述上表现出色，但在处理长而详细的描述时能力受限。现有方法借助区域建议信息将视觉区域与长描述中的句子对齐，但带来显著部署成本。我们提出MulCLIP，一种端到端的多层级对齐框架，将自然的长文本结构与图像组件有效对齐。首先，保持图像与摘要及长描述之间的全局对比对齐，并扩展位置嵌入以支持更长文本序列。为进一步提升细粒度理解，提出两项新策略：(1) 在局部校准特征上进行词元重建对齐，强化词语与图像块间的语义关联；(2) 子句聚合块对齐，自动提取并聚合每个子句对应的上下文丰富图像块。在多个基准上的实验表明，本方法持续提升下游性能，消融实验证实其多尺度对齐是优于区域建议方法的关键因素，特别适用于多样化的现实应用。
原文摘要 · Abstract (English)
Vision-language models like CLIP show impressive ability to align images and text, but their training on short, concise captions makes them struggle with lengthy, detailed descriptions. Recent advances mitigate this challenge by leveraging region-proposal information to map visual regions with corresponding sentences from lengthy captions, yet incurring notable deployment costs. We introduce MulCLIP, a novel end-to-end multi-level alignment framework that bridges natural long-text structures with image components. MulCLIP first preserves global contrastive alignment between images and both summary and long captions, while extending positional embeddings for longer text sequences. To further enhance fine-grained understanding, we propose two novel strategies: (1) a token reconstruction alignment over locally calibrated features to strengthen semantic connections between words and image patches, and (2) a subcaption-aggregated patch alignment that automatically extracts and aggregates context-rich patches for each subcaption. Experimental results across diverse benchmarks demonstrate our method consistently improves downstream performance, while ablation studies confirm its multi-scale alignment is the key factor driving better fine-grained capability than region-proposal-assisted approaches, making it particularly suitable for diverse real-world applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。