测试开源稀疏自编码器能否分离城市所属国家与洲的信息，发现效果不如神经元和线性方法。
Evaluating Open-Source Sparse Autoencoders on Disentangling Factual Knowledge in GPT-2 Small
- 用RAVEL基准评估GPT-2小模型的稀疏自编码器特征是否能独立表示国家与洲信息
- 四种开源SAE均未超越神经元基线，更远落后于分布式对齐搜索（DAS）方法
- 结果表明当前SAE特征空间在因果分析中仍不成熟，适合机制可解释性研究者参考
机制可解释性领域新流行的方法是训练高维稀疏自编码器（SAEs）以神经元激活为输入，并将SAE特征作为分析的基本单元。然而，关于SAE特征空间是否适用于因果分析的证据仍不充分。本文使用RAVEL基准，评估在GPT-2 small隐藏表示上训练的四个开源SAE，是否具备可分离城市所属国家与所在洲知识的特征集。以神经元为基线，以通过分布式对齐搜索（DAS）学习的线性特征为理想上限进行对比。针对每个模型，我们学习二值掩码以选择特征，实现仅改变城市国家而不影响洲的修复，或反之。结果表明，所有SAE均未达到神经元基线性能，更无法接近DAS表现。代码已开源：https://github.com/MaheepChaudhary/SAE-Ravel。
原文摘要 · Abstract (English)
A popular new method in mechanistic interpretability is to train high-dimensional sparse autoencoders (SAEs) on neuron activations and use SAE features as the atomic units of analysis. However, the body of evidence on whether SAE feature spaces are useful for causal analysis is underdeveloped. In this work, we use the RAVEL benchmark to evaluate whether SAEs trained on hidden representations of GPT-2 small have sets of features that separately mediate knowledge of which country a city is in and which continent it is in. We evaluate four open-source SAEs for GPT-2 small against each other, with neurons serving as a baseline, and linear features learned via distributed alignment search (DAS) serving as a skyline. For each, we learn a binary mask to select features that will be patched to change the country of a city without changing the continent, or vice versa. Our results show that SAEs struggle to reach the neuron baseline, and none come close to the DAS skyline. We release code here: https://github.com/MaheepChaudhary/SAE-Ravel
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。