arXiv:2508.16950cs.LGcs.CV2025-08

提出新指标,量化神经元多义性并验证其可解释性。

Disentangling Polysemantic Neurons with a Null-Calibrated Polysemanticity Index and Causal Patch Interventions

  • 设计零基准校准的多义性指数(PSI),融合几何聚类、类别对齐与CLIP语义区分。
  • 在ResNet-50上发现深层神经元多义性显著高于浅层,且激活集可分解为可命名原型。
  • 通过因果补丁干预验证:对齐补丁显著提升目标神经元激活,优于各类对照。

神经网络中的多义性神经元对多个甚至无关特征响应,阻碍机制可解释性。本文提出多义性指数(PSI),一种零基准校准的度量,用于判断神经元最高激活是否可分解为语义上独立的聚类。PSI由三个独立校准分量相乘构成:几何聚类质量(S)、与标注类别对齐度(Q)、以及通过CLIP实现的开放词汇语义区分度(D)。在预训练的ResNet-50模型上使用Tiny-ImageNet图像评估时,PSI识别出激活集能分解为语义连贯、可命名原型的神经元,并揭示显著的深度趋势:后期层神经元的PSI值远高于前期层。通过改变超参数、随机种子及跨编码器文本头进行鲁棒性检验,以及对比仅类别概念与开放词汇概念的广度分析,验证了方法可靠性。此外,采用因果补丁替换实验:对齐补丁显著提升目标神经元激活,优于非对齐、随机、位置打乱或全局掩码等对照组。因此,PSI为发现、量化和研究神经网络中的多义单元提供了原则性且实用的工具。

原文摘要 · Abstract (English)

Neural networks often contain polysemantic neurons that respond to multiple, sometimes unrelated, features, complicating mechanistic interpretability. We introduce the Polysemanticity Index (PSI), a null-calibrated metric that quantifies when a neuron's top activations decompose into semantically distinct clusters. PSI multiplies three independently calibrated components: geometric cluster quality (S), alignment to labeled categories (Q), and open-vocabulary semantic distinctness via CLIP (D). On a pretrained ResNet-50 evaluated with Tiny-ImageNet images, PSI identifies neurons whose activation sets split into coherent, nameable prototypes, and reveals strong depth trends: later layers exhibit substantially higher PSI than earlier layers. We validate our approach with robustness checks (varying hyperparameters, random seeds, and cross-encoder text heads), breadth analyses (comparing class-only vs. open-vocabulary concepts), and causal patch-swap interventions. In particular, aligned patch replacements increase target-neuron activation significantly more than non-aligned, random, shuffled-position, or ablate-elsewhere controls. PSI thus offers a principled and practical lever for discovering, quantifying, and studying polysemantic units in neural networks.

神经元可解释性多义性因果干预深度学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。