用自然语言重写大模型部分结构，保持性能同时提升可解释性。
Partially Rewriting a Transformer in Natural Language
- 用稀疏激活的MLP模拟原模型前馈层，生成自然语言解释。
- 替换第一层后模型损失增加接近全零替换，性能基本不变。
- 结果表明需更详细解释才能显著超越基础消融效果。
机械可解释性的终极目标是将深度神经网络完全重写为更易理解的形式，同时保持其行为与性能。本文尝试使用简单自然语言解释部分重写大型语言模型。首先，我们用一个更宽且稀疏激活的MLP（即transcoder）近似模型中的一个前馈网络，并通过自动化可解释性流程生成这些神经元的解释。接着，用基于LLM的模拟器替代该稀疏MLP的第一层，该模拟器根据解释和上下文预测每个神经元的激活状态。最后，我们测量这些修改对模型最终输出的影响。结果显示，模型损失的增长在统计上与完全用零向量替换稀疏MLP输出相当。我们采用相同方法，在同一层的残差流上使用稀疏自编码器，也得到相似结果。这表明，若想显著优于零消融基线，仍需要更详细的解释。
原文摘要 · Abstract (English)
The greatest ambition of mechanistic interpretability is to completely rewrite deep neural networks in a format that is more amenable to human understanding, while preserving their behavior and performance. In this paper, we attempt to partially rewrite a large language model using simple natural language explanations. We first approximate one of the feedforward networks in the LLM with a wider MLP with sparsely activating neurons - a transcoder - and use an automated interpretability pipeline to generate explanations for these neurons. We then replace the first layer of this sparse MLP with an LLM-based simulator, which predicts the activation of each neuron given its explanation and the surrounding context. Finally, we measure the degree to which these modifications distort the model's final output. With our pipeline, the model's increase in loss is statistically similar to entirely replacing the sparse MLP output with the zero vector. We employ the same protocol, this time using a sparse autoencoder, on the residual stream of the same layer and obtain similar results. These results suggest that more detailed explanations are needed to improve performance substantially above the zero ablation baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。