让CLIP同时关注全局与局部细节,提升视觉理解能力。
Helping CLIP See Both the Forest and the Trees: A Decomposition and Description Approach
- 用随机多裁剪增强激活模型局部特征感知能力
- 在零样本、少样本等场景下显著提升性能
- 无需修改模型结构,可直接插入现有流程
视觉语言模型(如CLIP)通过对比学习实现跨模态语义对齐,展现出强大的零样本泛化能力。然而,传统提示工程主要依赖粗粒度类别标签,忽略细粒度局部语义。现有方法假设模型天然具备局部细节识别能力,尝试通过大语言模型生成属性描述来增强文本提示。但我们系统实验发现:CLIP对全局图像模式存在强烈偏倚,难以处理局部视觉描述。为此,我们提出一种简单、高效且即插即用的方案,使CLIP能‘既见森林,也见树木’。具体而言,采用随机多裁剪增强,仅截取图像局部区域,有效限制模型感受野,重校准其注意力机制,缓解固有偏倚。我们在零样本、少样本及测试时适应设置下评估该方法,大量实验证明D&D表现优异。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) like CLIP achieve cross-modal semantic alignment through contrastive learning, exhibiting robust zero-shot generalization. Traditional prompt engineering, however, predominantly relies on coarse-grained category labels, neglecting fine-grained local semantics. Existing approaches assume that VLMs inherently recognize localized visual details and attempt to enhance classification by augmenting text prompts with attribute descriptors generated by large language models. However, our systematic experiments reveal critical limitations: CLIP's strong bias toward global image patterns hinders its ability to process localized visual descriptors. To address this fundamental constraint, we propose a simple, effective, and plug-and-play solution that enables CLIP to ``See Both the Forest and the Trees." Specifically, we employ stochastic multi-crop augmentation to activate CLIP's latent capacity for localized feature analysis. By cropping only partial regions, the approach effectively constrains the model's receptive field and recalibrates its attention mechanism, thereby mitigating its inherent bias. We evaluate the proposed method under zero-shot, few-shot, and test-time adaptation settings, and extensive experiments demonstrate that D&D achieves promising performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。