一个词典通用于多个大模型，解释力几乎不降。
SharedSAE: One Feature Dictionary Across Language Models
- 共享词典+模型专属编码器，跨模型复用潜在表示。
- 保留激活强度，单模型推理时用模型丢弃机制。
- 新模型可快速适配，适合需要统一解释的场景。
稀疏自编码器（SAE）广泛用于解释语言模型的激活，但传统方法需为每个模型单独训练并标注潜在变量。本文提出SharedSAE，使用单一共享词典搭配模型特定的编码器-解码器对。与之前方法不同，SharedSAE仅归一化选择分数，保留激活幅度，并通过模型丢弃实现单模型推理。我们在四个10亿参数规模、分属不同家族和分词器的基础语言模型上训练SharedSAE。尽管共享潜在空间，其平均解释方差仍达到专用SAE的96.6%；潜在激活在跨模型间的相关性是事后对齐独立SAE的1.8倍，且潜在语义可跨模型迁移。词典冻结后，新模型可高效适配，实现接近专用SAE的重建质量，并复用共享潜在描述。
原文摘要 · Abstract (English)
Sparse autoencoders (SAEs) are widely used to interpret language model activations, but SAE training and latent labelling are typically repeated for every model. Here, we show that a single shared SAE can replace a collection of dedicated per-model SAEs. Our method, SharedSAE, combines a shared dictionary with model-specific encoder-decoder pairs. Unlike the closest prior method, which discards activation magnitudes and requires all models at inference, SharedSAE instead normalizes only selection scores, preserving magnitudes, and uses model dropout for single-model inference. We train SharedSAE on four 1B-scale base language models spanning distinct families and tokenizers. Despite sharing its latents across models, SharedSAE retains 96.6% of dedicated SAEs' mean explained variance; its latent activations exhibit cross-model correlations 1.8 times as high as separate SAEs aligned post-hoc, and its latent descriptions transfer across models. After the dictionary is frozen, new models can be efficiently adapted to it, achieving near-dedicated-SAE reconstruction quality while reusing the shared latent descriptions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。