用扩散模型学习大模型内部状态分布，提升可解释性与干预效果。
Learning a Generative Meta-Model of LLM Activations
- 训练扩散模型学习十亿级残差流激活数据的分布，构建生成式元模型。
- 模型损失随算力下降，且损失越低，干预后文本流畅度提升越明显。
- 模型神经元更专注单一概念，适合追求无假设可解释性的研究者。
现有分析神经网络激活的方法（如PCA、稀疏自编码器）依赖强结构假设。生成模型提供替代方案：无需此类假设即可发现结构，并作为先验提升干预精度。本文在十亿个残差流激活上训练扩散模型，构建“元模型”以学习网络内部状态的分布。发现扩散损失随计算量平滑下降，且能可靠预测下游效用。将元模型学习到的先验用于引导干预时，文本流畅度提升，且收益随损失降低而增大。此外，元模型神经元逐渐将概念分离为独立单元，稀疏探测得分随损失降低而提升。结果表明，生成式元模型为无需限制性假设的可解释性提供了可扩展路径。
原文摘要 · Abstract (English)
Existing approaches for analyzing neural network activations, such as PCA and sparse autoencoders, rely on strong structural assumptions. Generative models offer an alternative: they can uncover structure without such assumptions and act as priors that improve intervention fidelity. We explore this direction by training diffusion models on one billion residual stream activations, creating "meta-models" that learn the distribution of a network's internal states. We find that diffusion loss decreases smoothly with compute and reliably predicts downstream utility. In particular, applying the meta-model's learned prior to steering interventions improves fluency, with larger gains as loss decreases. Moreover, the meta-model's neurons increasingly isolate concepts into individual units, with sparse probing scores that scale as loss decreases. These results suggest generative meta-models offer a scalable path toward interpretability without restrictive structural assumptions. Project page: https://generative-latent-prior.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。