破解神经网络内部工作机制，让模型决策可解释、可控制。
Mechanistic Interpretability for Neural Networks: Circuits, Sparse Features and Symbolic Reasoning
- 通过电路分析解析Transformer内部组件如何协同完成任务。
- 用稀疏自编码器分离出可理解的独立特征，解决激活纠缠问题。
- 将神经表征转化为逻辑规则，适合安全关键场景研究者使用。
本文系统综述了机制可解释性这一新兴领域，旨在逆向解析现代神经网络的内部算法。与仅停留在输入输出相关性的传统可解释AI不同，该方法直接应对机器学习模型的“黑箱”特性，对高风险部署中的安全性与可审计性至关重要。论文深入分析了Transformer电路，揭示残差流、注意力机制和归纳头如何驱动复杂任务与上下文学习。针对超叠加与多义性核心挑战，展示了稀疏自编码器（SAEs）和译码器如何将纠缠的网络激活分解为独立且可解释的特征。此外，论文还探讨了通过导向向量与因果干预主动操控模型行为的方法。最后，将这些机制洞察与神经符号人工智能框架结合，实现从神经表征到显式可执行逻辑规则的转换。
原文摘要 · Abstract (English)
This article offers a comprehensive overview of mechanistic interpretability, an emerging field that seeks to reverse-engineer the internal algorithms of modern neural networks. While traditional explainable AI methods often stop at surface-level input-output correlations, this approach directly addresses the opaque "black box" nature of machine learning models, which is essential for ensuring safety and auditability in high-stakes deployments. The paper provides a detailed examination of Transformer circuit analysis, exploring how internal components like the residual stream, attention mechanisms, and induction heads drive complex tasks and in-context learning. It subsequently tackles the core challenge of superposition and polysemanticity, demonstrating how tools like Sparse Autoencoders (SAEs) and transcoders can decompose tangled network activations into distinct, human-interpretable features. Furthermore, the paper explores methods for actively controlling and modifying model behavior through steering vectors and causal interventions. Finally, it connects these mechanistic insights with neurosymbolic AI frameworks designed to translate neural representations into explicit, executable logical rules.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。