arXiv:2511.00419cs.CVcs.AI2025-11被引 1

通过逐步扩展显著区域,提升视觉语言模型的语义表示能力。

LGCA: Enhancing Semantic Representation via Progressive Expansion

  • 先提取图像局部特征,再反复选择最显著区域进行扩展
  • 在多个数据集上实现零样本分类性能显著超越现有方法
  • 保持原模型复杂度,适合大规模应用

自然语言处理中的大规模预训练进展使得如CLIP等视觉语言模型能有效对齐图像与文本,在零样本图像分类任务中表现优异。后续研究发现,将图像裁剪为小区域并用大语言模型生成多描述,可进一步提升性能。然而,由于CLIP对随机裁剪敏感,小尺度下图像相似特征易引入错误信息和偏差。为此,本文提出局部-全局交叉对齐(LGCA)框架:先捕捉图像局部特征,再反复选择最显著区域并扩展。相似度计算融合原始与扩展图像信息,使模型同时捕获局部与全局特征,有效减少误导。此外,理论分析表明,该方法时间复杂度与原模型一致,具备高效性与可扩展性。大量实验验证其在多种数据集上显著提升零样本性能,优于当前最优基线。

原文摘要 · Abstract (English)

Recent advancements in large-scale pretraining in natural language processing have enabled pretrained vision-language models such as CLIP to effectively align images and text, significantly improving performance in zero-shot image classification tasks. Subsequent studies have further demonstrated that cropping images into smaller regions and using large language models to generate multiple descriptions for each caption can further enhance model performance. However, due to the inherent sensitivity of CLIP, random image crops can introduce misinformation and bias, as many images share similar features at small scales. To address this issue, we propose Localized-Globalized Cross-Alignment (LGCA), a framework that first captures the local features of an image and then repeatedly selects the most salient regions and expands them. The similarity score is designed to incorporate both the original and expanded images, enabling the model to capture both local and global features while minimizing misinformation. Additionally, we provide a theoretical analysis demonstrating that the time complexity of LGCA remains the same as that of the original model prior to the repeated expansion process, highlighting its efficiency and scalability. Extensive experiments demonstrate that our method substantially improves zero-shot performance across diverse datasets, outperforming state-of-the-art baselines.

视觉语言模型零样本学习特征对齐CLIP

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。