通过分解神经元与注意力头的贡献，让CLIP-ResNet的决策过程更可解释。
Interpreting ResNet-based CLIP via Neuron-Attention Decomposition
- 将神经元与注意力头配对，分解其在输出中的具体贡献路径。
- 仅少数神经元-注意力头对有显著影响，且能表示子概念。
- 无需训练即可实现语义分割，还可监测数据分布变化。
我们提出一种新方法，通过分解神经元在CLIP-ResNet中的贡献路径来解释其内部机制。具体而言，分析所有神经元与注意力池化层中注意力头的成对组合，发现这些神经元-注意力头对可在图像-文本嵌入空间中近似为单一方向。基于此，我们将每对关系与特定文本关联以实现解释。研究还发现，仅有稀疏的神经元-注意力头对对输出有显著贡献，且部分多义对代表对应神经元的子概念。利用这些发现，我们实现了两个应用：一是在不训练的情况下完成语义分割，性能优于先前方法；二是通过分析神经元-注意力头贡献动态监测数据分布偏移。结果表明，解析计算路径能揭示可解释单元，并可用于下游任务。
原文摘要 · Abstract (English)
We present a novel technique for interpreting the neurons in CLIP-ResNet by decomposing their contributions to the output into individual computation paths. More specifically, we analyze all pairwise combinations of neurons and the following attention heads of CLIP's attention-pooling layer. We find that these neuron-head pairs can be approximated by a single direction in CLIP-ResNet's image-text embedding space. Leveraging this insight, we interpret each neuron-head pair by associating it with text. Additionally, we find that only a sparse set of the neuron-head pairs have a significant contribution to the output value, and that some neuron-head pairs, while polysemantic, represent sub-concepts of their corresponding neurons. We use these observations for two applications. First, we employ the pairs for training-free semantic segmentation, outperforming previous methods for CLIP-ResNet. Second, we utilize the contributions of neuron-head pairs to monitor dataset distribution shifts. Our results demonstrate that examining individual computation paths in neural networks uncovers interpretable units, and that such units can be utilized for downstream tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。