让视觉语言模型的零样本分类结果可解释，通过局部对齐提升准确率和泛化性。
Interpretable Zero-Shot Learning with Locally-Aligned Vision-Language Model
- 用最优传输实现图像局部区域与语义属性的精准对齐
- 在多个数据集上显著提升零样本分类准确率并增强跨域泛化能力
- 无需额外训练即可获得可解释的预测过程，适合需要透明决策的场景
大规模视觉-语言模型（如CLIP）通过利用大规模图文配对数据，在零样本学习（ZSL）中取得了显著成功。然而，这些方法缺乏可解释性，因为它们将整个查询图像与类别词嵌入进行相似度计算，难以解释预测依据。为解决此问题，我们提出一种基于局部对齐的可解释零样本学习方法LaZSL。该方法通过最优传输实现视觉区域与对应语义属性之间的局部视觉-语义对齐，促进有效匹配并生成可解释的相似度，且无需额外训练。大量实验表明，该方法在可解释性、准确性及领域泛化能力方面均有显著提升。代码已开源：https://github.com/shiming-chen/LaZSL。
原文摘要 · Abstract (English)
Large-scale vision-language models (VLMs), such as CLIP, have achieved remarkable success in zero-shot learning (ZSL) by leveraging large-scale visual-text pair datasets. However, these methods often lack interpretability, as they compute the similarity between an entire query image and the embedded category words, making it difficult to explain their predictions. One approach to address this issue is to develop interpretable models by integrating language, where classifiers are built using discrete attributes, similar to human perception. This introduces a new challenge: how to effectively align local visual features with corresponding attributes based on pre-trained VLMs. To tackle this, we propose LaZSL, a locally-aligned vision-language model for interpretable ZSL. LaZSL employs local visual-semantic alignment via optimal transport to perform interaction between visual regions and their associated attributes, facilitating effective alignment and providing interpretable similarity without the need for additional training. Extensive experiments demonstrate that our method offers several advantages, including enhanced interpretability, improved accuracy, and strong domain generalization. Codes available at: https://github.com/shiming-chen/LaZSL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。