用模型自生成数据训练稀疏自编码器，提升特征可解释性稳定性。
FaithfulSAE: Towards Capturing Faithful Features with Sparse Autoencoders without External Dataset Dependencies
- 在模型自身生成的数据上训练SAE，避免外部数据偏差
- 跨种子实验更稳定，5/7模型假特征比例更低
- 适合关注大模型内部机制解释的研究者
稀疏自编码器（SAEs）被用于分解大语言模型表征以提取可解释特征。但近期研究指出，不同初始化下结果不稳定，且SAE可能无法捕捉模型内部特征。这可能源于在外部数据（如网络数据或另一模型生成数据）上训练，这些数据存在分布外（OOD）问题，导致生成‘假特征’，扭曲模型激活本质。为此，我们提出FaithfulSAE，通过在模型自身生成的合成数据上训练SAE来解决此问题。实验表明，在较少分布外指令数据上训练的FaithfulSAE在种子间更稳定；在探针任务中表现优于基于网络数据训练的SAE，且在7个模型中的5个中假特征比例更低。本方法消除了对外部数据集的依赖，推动了对模型内部特征的准确捕捉，并强调了训练数据选择的重要性。
原文摘要 · Abstract (English)
Sparse Autoencoders (SAEs) have emerged as a promising solution for decomposing large language model representations into interpretable features. However, Paulo and Belrose (2025) have highlighted instability across different initialization seeds, and Heap et al. (2025) have pointed out that SAEs may not capture model-internal features. These problems likely stem from training SAEs on external datasets - either collected from the Web or generated by another model - which may contain out-of-distribution (OOD) data beyond the model's generalisation capabilities. This can result in hallucinated SAE features, which we term "Fake Features", that misrepresent the model's internal activations. To address these issues, we propose FaithfulSAE, a method that trains SAEs on the model's own synthetic dataset. Using FaithfulSAEs, we demonstrate that training SAEs on less-OOD instruction datasets results in SAEs being more stable across seeds. Notably, FaithfulSAEs outperform SAEs trained on web-based datasets in the SAE probing task and exhibit a lower Fake Feature Ratio in 5 out of 7 models. Overall, our approach eliminates the dependency on external datasets, advancing interpretability by better capturing model-internal features while highlighting the often neglected importance of SAE training datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。