arXiv:2510.06541cs.CVcs.LG2025-10

通过聚类激活路径揭示神经网络决策逻辑,可识别错误依赖并检测异常输入。

Cluster Paths: Navigating Interpretability in Neural Networks

  • 将神经网络各层激活聚类,用簇编号序列表示输入,形成可读解释路径。
  • 在CelebA任务中达90%预测保真度,噪声下仍保持96%路径一致性。
  • 适用于大模型与小数据,能发现颜色、纹理等视觉概念,适合模型可信性研究者。

现代深度神经网络在视觉任务中表现优异,但其决策过程不透明,可能导致盲目信任、隐蔽偏见和意外失败。本文提出聚类路径(cluster paths),一种后处理可解释性方法:在选定层对激活进行聚类,并将每个输入表示为一系列簇编号。为此引入四项评估指标:路径复杂度(认知负荷)、加权路径纯度(类别对齐)、决策对齐忠实度(预测保真度)和路径一致性(扰动稳定性)。在含虚假线索的CIFAR-10实验中,聚类路径成功识别出基于颜色的捷径策略,并在该线索移除时路径坍缩;在五分类的CelebA发色任务中,实现90%忠实度,且在高斯噪声下保持96%一致性,同时不损失准确率。将方法扩展至ImageNet预训练的Vision Transformer,通过提示大型语言模型分析最小路径差异,生成概念路径。最后证明,聚类路径可有效作为分布外(OOD)检测器,在模型产生过度自信预测前可靠标记异常样本。该方法在多层网络中揭示了颜色、纹理或物体上下文等视觉概念,表明其可扩展至大型视觉模型,生成简洁且人类可读的解释。

原文摘要 · Abstract (English)

While modern deep neural networks achieve impressive performance in vision tasks, they remain opaque in their decision processes, risking unwarranted trust, undetected biases and unexpected failures. We propose cluster paths, a post-hoc interpretability method that clusters activations at selected layers and represents each input as its sequence of cluster IDs. To assess these cluster paths, we introduce four metrics: path complexity (cognitive load), weighted-path purity (class alignment), decision-alignment faithfulness (predictive fidelity), and path agreement (stability under perturbations). In a spurious-cue CIFAR-10 experiment, cluster paths identify color-based shortcuts and collapse when the cue is removed. On a five-class CelebA hair-color task, they achieve 90% faithfulness and maintain 96% agreement under Gaussian noise without sacrificing accuracy. Scaling to a Vision Transformer pretrained on ImageNet, we extend cluster paths to concept paths derived from prompting a large language model on minimal path divergences. Finally, we show that cluster paths can serve as an effective out-of-distribution (OOD) detector, reliably flagging anomalous samples before the model generates over-confident predictions. Cluster paths uncover visual concepts, such as color palettes, textures, or object contexts, at multiple network depths, demonstrating that cluster paths scale to large vision models while generating concise and human-readable explanations.

可解释性神经网络视觉模型路径分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。