提出一种新方法，在持续学习中几乎零遗忘地扩展模型能力。
Gate-Zero Growth: A Geometric Framework for Function-Preserving Continual Learning
- 用零初始化门控添加残差块，保持旧功能不变。
- 实验显示旧任务遗忘低于0.1，远优于传统方法。
- 适合需要安全扩容的持续学习场景。
我们提出一种函数保持（FP）的持续学习算子——门零增长（gate-zero growth），通过零初始化门控添加新的残差块。在横截条件成立时，该方法在函数雅可比矩阵中诱导出秩分离：旧方向保持不变，新权重方向在增长点恰好平坦，而新门控方向是唯一的一阶新功能来源。随着门控开启，函数漂移为$O(oldsymbolα^2)$，雅可比泄漏为$O(oldsymbolα_ ext{∞})$，实现对函数保持轨迹的受控偏离。在从WikiText-103迁移到BookCorpus的300M至857M Transformer上，门零增长在精确保持（隔离）和联合前沿（无冻结）两种设定下均实现近零旧域遗忘（Δ_A < 0.1），而对比的非函数保持控制方法（G_{ ext{stack}}）遗忘量大一个数量级。相同的几何分析覆盖了LoRA、ReZero和零初始化适配器结构，确立门零增长为持续学习中安全容量激活的通用局部几何范式。
原文摘要 · Abstract (English)
We introduce \emph{gate-zero growth}, a function-preserving (FP) operator for continual learning that adds new residual blocks through a zero-initialised gate. Under a transversality condition, gate-zero growth induces \emph{rank separation} in the functional Jacobian: old directions are unchanged, new-weight directions are exactly flat at the growth point, and new gate directions are the only first-order source of new functional variation. As gates open during continual learning, function drift is $O(\|\boldsymbolα\|^2)$ and Jacobian leakage $O(\|\boldsymbolα\|_\infty)$, giving a controlled departure from the FP locus. On a $300\mathrm{M}\to857\mathrm{M}$ Transformer adapted from WikiText-103 to BookCorpus, gate-zero growth reaches near-zero old-domain forgetting ($Δ_A < 0.1$) under both exact-preservation (Isolation) and joint-frontier (Freeze-Nothing) operating points, while a non-FP control ($G_{\text{stack}}$) suffers an order-of-magnitude larger forgetting under the same recipe. The same geometric analysis covers LoRA, ReZero, and zero-init adapter constructions, establishing gate-zero growth as the canonical instance of a shared local geometry that governs safe capacity activation in CL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。