发现并解释神经网络中缺失概念引发的激活,提升模型可解释性
What is Missing? Explaining Neurons Activated by Absent Concepts
- 提出扩展方法,识别输入缺失时反而激活神经元的现象
- 实验证明图像分类模型常利用缺失概念进行判断
- 对去偏和可解释性提升有实际价值,适合研究模型偏差者
可解释人工智能(XAI)旨在提供深度神经网络行为的人类可理解洞察,通常通过估计模型的简化因果结构实现。现有方法多关注概念存在与神经元强激活之间的关联,如基于梯度的归因方法识别影响预测的关键输入像素,特征可视化揭示使目标神经元高激活的输入——前者隐含信息存在于输入中,后者认为神经元编码了概念的存在。然而,一种被广泛忽视的因果关系是:概念缺失反而导致神经元激活。本文表明,这种编码缺失的现象普遍存在,而主流XAI方法在标准形式下难以揭示。为此,我们提出对归因与特征可视化技术的两项简单扩展,以发现并解释编码缺失。实验显示,主流XAI方法可有效揭示编码缺失,且ImageNet模型普遍利用此类模式;考虑缺失信息还能改善去偏效果。
原文摘要 · Abstract (English)
Explainable artificial intelligence (XAI) aims to provide human-interpretable insights into the behavior of deep neural networks (DNNs), typically by estimating a simplified causal structure of the model. In existing work, this causal structure often includes relationships where the presence of a concept is associated with a strong activation of a neuron. For example, attribution methods primarily identify input pixels that contribute most to a prediction, and feature visualization methods reveal inputs that cause high activation of a target neuron - the former implicitly assuming that the relevant information resides in the input, and the latter that neurons encode the presence of concepts. However, a largely overlooked type of causal relationship is that of encoded absences, where the absence of a concept increases neural activation. In this work, we show that such missing but relevant concepts are common and that mainstream XAI methods struggle to reveal them when applied in their standard form. To address this, we propose two simple extensions to attribution and feature visualization techniques that uncover encoded absences. Across experiments, we show how mainstream XAI methods can be used to reveal and explain encoded absences, how ImageNet models exploit them, and that debiasing can be improved when considering them.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。