提出高效提取大模型稀疏电路的方法，解决计算复杂度与特征复用难题。
Hierarchical Sparse Circuit Extraction from Billion-Parameter Language Models through Scalable Attribution Graph Decomposition
- 分层图分解技术降低搜索复杂度至O(n²log n)
- 在多个模型上实现91%行为保留，电路规模49-347节点
- 适合研究模型可解释性与跨架构迁移的学者
从十亿参数量级的Transformer模型中提取稀疏电路面临计算开销$O(2^n)$和路径间广泛特征复用的挑战。本文提出分层归因图分解（HAGD），包含四个阶段：跨层编码器训练、归因图谱的谱粗化、基于注意力GNN的分层遍历，以及因果干预验证，将最坏情况复杂度降至$O(n^2 \log n)$。基于RedPajama语料训练的每层编码器生成单义词典；梯度-激活乘积构建加权归因图；归一化拉普拉斯谱聚类建立多分辨率层次结构；注意力GNN在逐级粗化阶段分配电路成员得分。评估覆盖GPT-2（117M-774M）、Pythia（1.4B-6.9B）和Llama（7B-70B），任务包括模运算、奇偶校验、整数排序、共指消解（WinoGrande）、常识推理（HellaSwag）和事实记忆。在模运算任务上，行为保留率达91%（±2.3%），对应电路规模49-347节点；而传统ACDC方法在超过1.4B参数时内存耗尽。跨架构迁移系数为0.38-0.82，同家族对（如Llama-7B→Llama-70B）达0.82。局限包括忽略注意力头电路、15-20%未解释重构方差、验证过程存在回环问题，以及数百节点以上电路的可解释性存疑。
原文摘要 · Abstract (English)
Extracting sparse circuits from billion-parameter transformers is constrained by $O(2^n)$ search cost and pervasive feature reuse across co-active pathways. Hierarchical Attribution Graph Decomposition (HAGD) addresses this through four stages: cross-layer transcoder training, spectral coarsening of attribution graphs, graph-neural-network (GNN)-guided hierarchical traversal, and causal intervention verification, reducing worst-case complexity to $O(n^2 \log n)$. Per-layer transcoders trained on the RedPajama corpus yield monosemantic dictionaries; gradient-activation products form weighted attribution graphs; normalized-Laplacian spectral clustering builds multi-resolution hierarchies; an attention-based GNN assigns circuit-membership scores at successive coarsening stages. Evaluation spans GPT-2 (117M-774M), Pythia (1.4B-6.9B), and Llama (7B-70B) across modular arithmetic, parity computation, integer sorting, coreference resolution (WinoGrande), commonsense reasoning (HellaSwag), and factual recall. Behavioral preservation reaches 91\% ($\pm$2.3\%) on modular arithmetic with 49-347-node circuits, while ACDC exhausts memory beyond 1.4B parameters. Cross-architecture transfer coefficients span 0.38-0.82, with within-family pairs (Llama-7B $\to$ Llama-70B) attaining 0.82. Limitations include omitted attention-head circuits, 15-20\% unexplained reconstruction variance, ablation-based validation circularity, and uncertain interpretability of circuits exceeding several hundred nodes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。