提升图像模型对细粒度局部特征的感知,解决零样本组合识别难题
MSCI: Addressing CLIP's Inherent Limitations for Compositional Zero-Shot Learning
- 分阶段融合低层局部与高层全局视觉特征
- 在三个数据集上准确率显著超越现有方法
- 适合需要灵活适应复杂组合场景的研究者
组合零样本学习(CZSL)旨在通过已知的属性-物体组合来识别未见的组合。现有方法主要依赖CLIP的跨模态对齐能力,但忽略了其在捕捉细粒度局部特征方面的固有局限,这源于其架构和训练范式。为此,我们提出多阶段跨模态交互(MSCI)模型,有效利用CLIP视觉编码器中间层信息。具体地,设计两个自适应聚合器,分别从低层视觉特征提取局部信息、从高层特征整合全局信息,并通过逐阶段交互机制逐步融入文本表示,显著增强模型对细粒度局部视觉信息的感知能力。此外,MSCI根据不同组合及组合内元素动态调整全局与局部信息的注意力权重,实现对多样化场景的灵活适应。在三个常用数据集上的实验充分验证了该模型的有效性与优越性。代码与数据已公开于https://github.com/ltpwy/MSCI。
原文摘要 · Abstract (English)
Compositional Zero-Shot Learning (CZSL) aims to recognize unseen state-object combinations by leveraging known combinations. Existing studies basically rely on the cross-modal alignment capabilities of CLIP but tend to overlook its limitations in capturing fine-grained local features, which arise from its architectural and training paradigm. To address this issue, we propose a Multi-Stage Cross-modal Interaction (MSCI) model that effectively explores and utilizes intermediate-layer information from CLIP's visual encoder. Specifically, we design two self-adaptive aggregators to extract local information from low-level visual features and integrate global information from high-level visual features, respectively. These key information are progressively incorporated into textual representations through a stage-by-stage interaction mechanism, significantly enhancing the model's perception capability for fine-grained local visual information. Additionally, MSCI dynamically adjusts the attention weights between global and local visual information based on different combinations, as well as different elements within the same combination, allowing it to flexibly adapt to diverse scenarios. Experiments on three widely used datasets fully validate the effectiveness and superiority of the proposed model. Data and code are available at https://github.com/ltpwy/MSCI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。