实现神经网络手术的精确插入,保持功能不变且参数立即可训练。
Exact Network Surgery: Functional Invariance and Gradient Plasticity in Reactive Computational Graphs
- 在运行中精确插入残差块,函数输出完全一致,无需重训练。
- 插入后梯度门立即获得非零梯度,激活新分支学习能力。
- 适用于需要动态扩展模型的实时系统,如在线学习场景。
函数保持型网络增长技术(如Net2Net和渐进堆叠)可在不破坏已学功能的前提下扩展模型容量,但现有方法或容忍数值误差,或需完整重建训练程序。本文提出精确网络手术:在运行中的计算图内原位插入残差块,保证(1)网络函数精确保留——在显式浮点假设下逐位一致;(2)插入参数可立即训练。证明了门控残差块的身份同构定理、结构局部性定理(反应式失效引擎仅重计算插入点下游锥体,其余节点值与优化器状态不变),以及逃逸初始化命题(梯度遮蔽门alpha初始为零,在插入时获得一般非零梯度)。识别出一种退化配置——零初始化输出投影结合零门——是梯度下降无法逃离的精确鞍点。所有结论均在Julia中的NeuroDSL反应式图引擎参考实现中验证:1600个logit测试无一错;门在第一个优化步即逃逸零值,第二个步解锁分支梯度,完全符合预测;退化配置在整个600步运行中梯度恒为零;手术开销与下游锥体大小高度相关(r=0.9992),而插入+失效管理开销恒定(约0.75毫秒);真实重启后训练可逐位复现。附录报告首次单种子观察到插入后门动态。
原文摘要 · Abstract (English)
Function-preserving network growth techniques such as Net2Net and progressive stacking expand a model's capacity without destroying its learned function, but existing formulations either tolerate numerical perturbations or require a full rebuild of the training program. We formalize Exact Network Surgery: the in-place insertion of a residual block into a live computational graph such that (i) the network function is preserved -- bit-exactly under explicit floating-point hypotheses -- and (ii) inserted parameters remain trainable immediately after insertion. We prove an identity-morphism theorem for gated residual blocks, a structural-locality theorem showing that a reactive invalidation engine recomputes exactly the downstream cone of the insertion point, leaving every other node's value and optimizer state untouched, and an escape-from-initialization proposition showing that the Gradient Shadowing gate alpha, initialized at zero over a randomly initialized branch, receives a generically non-zero gradient at insertion time. We identify a degenerate configuration -- zero-initialized output projections combined with a zero gate -- that is an exact saddle point gradient descent cannot escape. Every claim is validated on the reference implementation in NeuroDSL, a reactive graph engine in Julia: grafting is bit-exact on every logit tested (0 mismatches out of 1600); the gate escapes zero at the first optimizer step and unlocks branch gradients at the second, exactly as predicted; the degenerate configuration exhibits gradients identically zero for the entire 600-step run; surgery cost tracks downstream cone size with r = 0.9992 while graft-plus-invalidation bookkeeping is constant (about 0.75 ms) across insertion depths; and training resumes bit-identically across a real process restart. A flagged preliminary appendix reports first single-seed observations on post-insertion gate dynamics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。