将机制可解释性视为统计估计,揭示其结果不稳定的根本原因。
Mechanistic Interpretability as Statistical Estimation: A Variance Analysis
- 把电路发现看作基于因果中介分析的统计估计问题
- 单输入因果效应评分具有高内在方差,结果不可靠
- 建议报告稳定性指标,提升可解释性研究的严谨性
机制可解释性(MI)旨在通过识别功能子网络来逆向解析模型行为。然而,这些发现的科学有效性取决于其稳定性。本文认为,电路发现并非独立任务,而是建立在因果中介分析(CMA)之上的统计估计问题。我们发现该基础层存在根本性不稳定性:精确的单输入CMA评分具有高内在方差,表明组件的因果效应是易变的随机变量而非固定属性。随后我们证明,电路发现流程继承并放大了这种方差。快速近似方法(如边缘归因修补及其后续方法)引入额外估计噪声,而对数据集上这些有噪评分的聚合则导致结构估计脆弱。微小的输入数据或超参数扰动即引发截然不同的电路结果。我们系统分解了各类方差来源,倡导更严格的MI实践,优先考虑统计稳健性,并常规报告稳定性度量。
原文摘要 · Abstract (English)
Mechanistic Interpretability (MI) aims to reverse-engineer model behaviors by identifying functional sub-networks. Yet, the scientific validity of these findings depends on their stability. In this work, we argue that circuit discovery is not a standalone task but a statistical estimation problem built upon causal mediation analysis (CMA). We uncover a fundamental instability at this base layer: exact, single-input CMA scores exhibit high intrinsic variance, implying that the causal effect of a component is a volatile random variable rather than a fixed property. We then demonstrate that circuit discovery pipelines inherit this variance and further amplify it. Fast approximation methods, such as Edge Attribution Patching and its successors, introduce additional estimation noise, while aggregating these noisy scores over datasets leads to fragile structural estimates. Consequently, small perturbations in input data or hyperparameters yield vastly different circuits. We systematically decompose these sources of variance and advocate for more rigorous MI practices, prioritizing statistical robustness and routine reporting of stability metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。