arXiv:2501.16374cs.LGcs.AI2025-01NAACL被引 2

通过神经元重分配提升大模型可解释性,不牺牲准确率。

SAFR: Neuron Redistribution for Interpretability

  • 用损失函数正则化,让关键词对应单一功能神经元
  • 相关词对保留多义性,整体保持高预测准确率
  • 可视化中间层神经元分配,提供可解释依据

超叠加(Superposition)指单个神经元编码多个特征表示,常见于深度神经网络。该特性使模型能组合并表达复杂信息,但降低了可解释性。本文提出SAFR方法,通过在损失函数中引入正则化,促进重要标记的单义性表示,同时鼓励相关标记对的多义性表示。重要标记和相关标记对通过VMASK和注意力权重识别。在两个分类任务上使用Transformer模型验证,实验表明SAFR在不损害预测性能的前提下显著提升模型可解释性。此外,SAFR可通过可视化中间层神经元分配提供解释。

原文摘要 · Abstract (English)

Superposition refers to encoding representations of multiple features within a single neuron, which is common in deep neural networks. This property allows neurons to combine and represent multiple features, enabling the model to capture intricate information and handle complex tasks. Despite promising performance, the model's interpretability has been diminished. This paper presents a novel approach to enhance model interpretability by regularizing feature superposition. We introduce SAFR, which simply applies regularizations to the loss function to promote monosemantic representations for important tokens while encouraging polysemanticity for correlated token pairs, where important tokens and correlated token pairs are identified via VMASK and attention weights respectively. We evaluate SAFR with a transformer model on two classification tasks. Experiments demonstrate the effectiveness of SAFR in improving model interpretability without compromising prediction performance. Besides, SAFR provides explanations by visualizing the neuron allocation within the intermediate layers.

可解释性神经元分配Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。