arXiv:2503.18483cs.CV2025-03CVPR被引 5

用语言提示消除视觉领域偏差,提升模型泛化能力

Explaining Domain Shifts in Language: Concept erasing for Interpretable Image Classification

  • 通过语言模型生成未见领域的描述符,模拟视觉领域偏移
  • 引入正交性正则项,显著提升跨域泛化性能
  • 适用于高风险场景,增强可解释性与鲁棒性

概念模型可将黑箱表示映射为人类可理解的概念,使决策过程更透明,便于理解预测依据。然而,特定领域的概念常影响最终预测,削弱模型泛化能力,限制其在高风险场景的应用。本文提出一种新型语言引导概念擦除框架(LanCE)。实证表明,预训练视觉语言模型(VLMs)可通过领域描述符近似不同视觉领域偏移,而大语言模型(LLMs)可轻松生成大量未见视觉领域的描述符。为此,我们引入一种无需依赖概念模型设计的插件式领域描述符正交性(DDO)正则项,有效缓解领域特定概念对预测的影响。在四个标准基准和三个新引入基准上评估,结果显示DDO显著优于现有最先进概念模型,在分布外(OOD)泛化方面表现突出。代码已开源。

原文摘要 · Abstract (English)

Concept-based models can map black-box representations to human-understandable concepts, which makes the decision-making process more transparent and then allows users to understand the reason behind predictions. However, domain-specific concepts often impact the final predictions, which subsequently undermine the model generalization capabilities, and prevent the model from being used in high-stake applications. In this paper, we propose a novel Language-guided Concept-Erasing (LanCE) framework. In particular, we empirically demonstrate that pre-trained vision-language models (VLMs) can approximate distinct visual domain shifts via domain descriptors while prompting large Language Models (LLMs) can easily simulate a wide range of descriptors of unseen visual domains. Then, we introduce a novel plug-in domain descriptor orthogonality (DDO) regularizer to mitigate the impact of these domain-specific concepts on the final predictions. Notably, the DDO regularizer is agnostic to the design of concept-based models and we integrate it into several prevailing models. Through evaluation of domain generalization on four standard benchmarks and three newly introduced benchmarks, we demonstrate that DDO can significantly improve the out-of-distribution (OOD) generalization over the previous state-of-the-art concept-based models.Our code is available at https://github.com/joeyz0z/LanCE.

可解释性领域泛化视觉语言模型概念删除

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。