发现大模型前馈层中存在关键通道核心,保护它可显著提升剪枝效果。
Supernodes and Halos: Loss-Critical Hubs in LLM Feed-Forward Layers

- 用损失敏感度指标定位每层最核心的1%通道(称超节点)
- 剪枝时保护这些核心通道,模型困惑度仅从989降至54.8
- 该结构在多个大模型中普遍存在,适合高效模型压缩研究
我们研究了Transformer前馈网络(FFN)中通道级重要性的组织结构。基于激活梯度二阶矩的损失代理(LP),发现损失敏感度集中在每层少数通道中。在Llama-3.1-8B中,每层前1%的通道贡献了中位数58.7%的LP质量,范围为33.0%至86.1%,这些通道被称为超节点。尽管FFN层也存在强激活异常值,但超节点与激活定义的异常值重叠微弱,且无法仅由激活强度或权重范数解释。围绕这一核心,我们发现一个较弱但一致的晕环结构:部分非超节点通道共享核心的写入支持,表现出更强冗余性。通过单次结构化剪枝测试该结构。在50%的FFN稀疏度下,剪掉大量超节点的基线方法性能急剧下降,而我们提出的SCAR变体显式保护超节点核心;其中最强的SCAR-Prot版本达到困惑度54.8,相较之下Wanda-channel为989.2。该LP集中模式在Mistral-7B、Llama-2-7B和Qwen2-7B中均出现,即使在目标的Llama-3.1-70B实验中仍可见,并在OLMo-2-7B预训练过程中增强。结果表明,大语言模型的FFN会发展出一个小的、学习到的核心损失敏感通道,保护此核心对可靠结构化剪枝至关重要。
原文摘要 · Abstract (English)
We study the organization of channel-level importance in transformer feed-forward networks (FFNs). Using a Fisher-style loss proxy (LP) based on activation-gradient second moments, we show that loss sensitivity is concentrated in a small set of channels within each layer. In Llama-3.1-8B, the top 1% of channels per layer accounts for a median of 58.7% of LP mass, with a range of 33.0% to 86.1%. We call these loss-critical channels supernodes. Although FFN layers also contain strong activation outliers, LP-defined supernodes overlap only weakly with activation-defined outliers and are not explained by activation power or weight norms alone. Around this core, we find a weaker but consistent halo structure: some non-supernode channels share the supernodes' write support and show stronger redundancy with the protected core. We use one-shot structured FFN pruning as a diagnostic test of this organization. At 50% FFN sparsity, baselines that prune many supernodes degrade sharply, whereas our SCAR variants explicitly protect the supernode core; the strongest variant, SCAR-Prot, reaches perplexity 54.8 compared with 989.2 for Wanda-channel. The LP-concentration pattern appears across Mistral-7B, Llama-2-7B, and Qwen2-7B, remains visible in targeted Llama-3.1-70B experiments, and increases during OLMo-2-7B pretraining. These results suggest that LLM FFNs develop a small learned core of loss-critical channels, and that preserving this core is important for reliable structured pruning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。