发现随机初始化的Transformer也能获得高可解释性评分
Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformers
- 用稀疏自编码器检测随机初始化Transformer的可解释性
- 随机模型在多数情况下表现与训练好的模型相近
- 提醒别只靠分数判断模型是否真正学到有用特征
稀疏自编码器（SAEs）广泛用于从Transformer激活中提取稀疏且可解释的潜在表示。我们测试了常用的SAE质量指标和自动解释流程，能否区分训练过的Transformer与随机初始化的模型（参数独立同分布采样自高斯）。在多种Pythia模型规模和随机化方案下，我们发现，在许多情况下，对随机初始化模型训练的SAE所得到的自动可解释性得分和重建性能，与训练模型相当。这表明，高整体可解释性分数本身并不能保证已恢复出学习到的、具有计算意义的特征。因此我们建议将常见SAE指标视为有用但不充分的可解释性代理，并主张常规使用随机基线和针对性衡量特征‘抽象性’的方法。
原文摘要 · Abstract (English)
Sparse autoencoders (SAEs) are widely used to extract sparse, interpretable latents from transformer activations. We test whether commonly used SAE quality metrics and automatic explanation pipelines can distinguish trained transformers from randomly initialized ones (e.g., where parameters are sampled i.i.d. from a Gaussian). Over a wide range of Pythia model sizes and multiple randomization schemes, we find that, in many settings, SAEs trained on randomly initialized transformers produce auto-interpretability scores and reconstruction metrics that are similar to those from trained models. These results show that high aggregate auto-interpretability scores do not, by themselves, guarantee that learned, computationally relevant features have been recovered. We therefore recommend treating common SAE metrics as useful but insufficient proxies for mechanistic interpretability and argue for routine randomized baselines and targeted measures of feature 'abstractness'.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。