arXiv:2603.08424cs.LGcs.AI2026-03

提出无需重训练的框架,分析Transformer模型内部神经元行为与脆弱性。

SYNAPSE: Framework for Neuron Analysis and Perturbation in Sequence Encoding

  • 通过提取[CLS]表示并用轻量线性探针排序神经元,实现无损干预分析。
  • 发现任务信息由重叠神经元子集编码,具强功能稳定性但存在标签敏感差异。
  • 可识别模型弱点,指导构建更鲁棒的Transformer模型,适合安全敏感场景。

近年来,人工智能在数据分析、预测和问题解决等复杂任务中表现强大,但其缺乏透明性引发可靠性担忧。在医疗或网络安全等敏感领域,确保透明性、可信度和鲁棒性至关重要,因错误决策或攻击后果严重。现有神经元层面可解释性方法多为描述性、任务依赖或需重新训练,难以作为系统化、可复用的工具评估跨架构与领域的内部鲁棒性。为此,本文提出SYNAPSE——一种系统性、无需训练的框架,用于理解并压力测试Transformer模型在不同领域的内部行为。该框架提取每层[CLS]表示,训练轻量线性探针获得全局与类别级神经元排名,并在推理阶段应用前向钩子干预。此设计使模型内部表示可在不修改原模型的前提下进行受控实验,从而直接测量、比较各任务与架构中的弱点、稳定性模式与标签特异性敏感性。所有实验揭示了一种一致的、与领域无关的内部表示组织:任务相关信息由广泛重叠的神经元子集编码。这种冗余带来强功能性稳定,而类别间的不对称性暴露异质化专业化模式,支持标签感知分析。相反,权重或logit空间的小型结构化扰动即可引导预测方向,凸显互补的脆弱性特征,表明SYNAPSE能有效指导更鲁棒Transformer模型的开发。

原文摘要 · Abstract (English)

In recent years, Artificial Intelligence has become a powerful partner for complex tasks such as data analysis, prediction, and problem-solving, yet its lack of transparency raises concerns about its reliability. In sensitive domains such as healthcare or cybersecurity, ensuring transparency, trustworthiness, and robustness is essential, since the consequences of wrong decisions or successful attacks can be severe. Prior neuron-level interpretability approaches are primarily descriptive, task-dependent, or require retraining, which limits their use as systematic, reusable tools for evaluating internal robustness across architectures and domains. To overcome these limitations, this work proposes SYNAPSE, a systematic, training-free framework for understanding and stress-testing the internal behavior of Transformer models across domains. It extracts per-layer [CLS] representations, trains a lightweight linear probe to obtain global and per-class neuron rankings, and applies forward-hook interventions during inference. This design enables controlled experiments on internal representations without altering the original model, thereby allowing weaknesses, stability patterns, and label-specific sensitivities to be measured and compared directly across tasks and architectures. Across all experiments, SYNAPSE reveals a consistent, domain-independent organization of internal representations, in which task-relevant information is encoded in broad, overlapping neuron subsets. This redundancy provides a strong degree of functional stability, while class-wise asymmetries expose heterogeneous specialization patterns and enable label-aware analysis. In contrast, small structured manipulations in weight or logit space are sufficient to redirect predictions, highlighting complementary vulnerability profiles and illustrating how SYNAPSE can guide the development of more robust Transformer models.

神经元分析Transformer可解释性鲁棒性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。