提出统一框架,让神经网络剪枝后仍保持计算功能并可验证因果抽象。
Causal Mechanism Reduction: Mechanism Replacement for Neural Network Pruning and Abstraction
- 将神经网络视为因果模型,用常数或仿射函数替换内部变量以实现剪枝。
- 新方法在ImageNet-100上与VBP性能相当,且在因果干预测试中更稳定。
- 适合关注模型压缩、可解释性及因果验证的研究者。
神经网络哪些内部机制可被替换而不影响其计算?结构化剪枝旨在获得更小的可部署模型;因果抽象则追求与干预操作兼容的高层模型。本文提出因果机制约简(CMR)框架,将训练好的网络视为确定性结构因果模型,用常数或保留变量的仿射函数替换选定的内部变量。此类替换可通过偏置与权重折叠精确编译为更小的稠密网络,并生成可通过对换干预测试的简化因果模型。我们推导出统一的二阶替换风险目标,其特例涵盖均值替换、基于方差的剪枝(VBP)、logit失真评分和仿射神经元合并,并给出基于边界的安全证明确保logit失真与对换干预一致性。该框架还揭示一个基本不变性要求:功能等价的ReLU网络应产生相同约简结果。在精确正缩放重参数化下,VBP的保留集退化为随机水平重叠,而logit失真评分完全不变。实验表明,CMR变体在DeiT-Tiny于ImageNet-100上经匹配微调后表现媲美VBP;更显著差异出现在不变性与对换测试中,其中logit失真评分保持保留集一致并持续提升分布保真度。因此,CMR为剪枝、编译与因果抽象验证提供了一个共同优化与验证对象。
原文摘要 · Abstract (English)
Which internal mechanisms of a neural network can be replaced while preserving the computation it performs? Structured pruning asks for smaller deployable networks; causal abstraction asks for high-level models that commute with interventions. We introduce causal mechanism reduction (CMR), a framework that treats a trained network as a deterministic structural causal model and replaces selected internal variables by constants or affine functions of retained variables. These replacements compile exactly into smaller dense networks by bias and weight folding, and induce reduced causal models testable with interchange interventions. We derive a unified second-order replacement-risk objective whose special cases recover mean replacement, variance-based pruning (VBP), logit-distortion scoring, and affine neuron merging, together with a margin-based certificate linking logit distortion to interchange-intervention agreement. The framework also exposes a basic invariance requirement: functionally identical ReLU networks should induce the same reduction. Under exact positive-scaling reparameterizations, VBP's kept set collapses to chance-level overlap while the logit-distortion score is exactly invariant. Empirically, CMR variants are competitive with VBP under matched fine-tuning of DeiT-Tiny on ImageNet-100; the clearer separation appears in the invariance and interchange tests, where the logit-distortion score preserves kept sets and consistently improves distributional fidelity. CMR thus gives pruning, compilation, and causal-abstraction verification a common object to optimize and verify.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。