稀疏自编码器无法找到唯一的原子特征,其发现的特征既不完整也不原子。
Sparse Autoencoders Do Not Find Canonical Units of Analysis
- 通过拼接大模型潜空间,发现小模型遗漏新信息,证明不完整
- 用元自编码器分析发现特征可分解为更小特征组合,非原子单位
- 适合关注特征可解释性但不追求唯一性的研究者使用
机制可解释性的常见目标是将神经网络激活分解为特征:由模型计算的输入可解释属性。稀疏自编码器(SAEs)是用于在大语言模型中寻找这些特征的流行方法,有人认为它们能发现一组‘规范’的原子特征。本文通过两种新方法质疑这一观点:一是通过SAE拼接显示其不完整,二是通过元SAE显示其非原子性。拼接实验表明,从大SAE引入的潜变量可分为‘新特征’和‘重建特征’,前者提升性能说明小SAE遗漏新信息;元SAE训练于另一SAE的解码器矩阵,发现大SAE的潜变量常可分解为更小潜变量的组合,如‘爱因斯坦’可拆解为‘科学家’‘德国人’‘名人’等。这表明大SAE的特征并非原子。即使无法找到规范单元,SAEs仍具实用价值。建议未来研究或另寻路径识别此类单元,或根据任务需求选择合适规模的SAE。我们提供交互式元SAE探索界面:https://metasaes.streamlit.app/
原文摘要 · Abstract (English)
A common goal of mechanistic interpretability is to decompose the activations of neural networks into features: interpretable properties of the input computed by the model. Sparse autoencoders (SAEs) are a popular method for finding these features in LLMs, and it has been postulated that they can be used to find a \textit{canonical} set of units: a unique and complete list of atomic features. We cast doubt on this belief using two novel techniques: SAE stitching to show they are incomplete, and meta-SAEs to show they are not atomic. SAE stitching involves inserting or swapping latents from a larger SAE into a smaller one. Latents from the larger SAE can be divided into two categories: \emph{novel latents}, which improve performance when added to the smaller SAE, indicating they capture novel information, and \emph{reconstruction latents}, which can replace corresponding latents in the smaller SAE that have similar behavior. The existence of novel features indicates incompleteness of smaller SAEs. Using meta-SAEs -- SAEs trained on the decoder matrix of another SAE -- we find that latents in SAEs often decompose into combinations of latents from a smaller SAE, showing that larger SAE latents are not atomic. The resulting decompositions are often interpretable; e.g. a latent representing ``Einstein'' decomposes into ``scientist'', ``Germany'', and ``famous person''. Even if SAEs do not find canonical units of analysis, they may still be useful tools. We suggest that future research should either pursue different approaches for identifying such units, or pragmatically choose the SAE size suited to their task. We provide an interactive dashboard to explore meta-SAEs: https://metasaes.streamlit.app/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。