新方法让模型适应新任务时不遗忘旧知识，还能高效复用预训练能力。
Grow, Don't Overwrite: Fine-tuning Without Forgetting
- 通过复制参数并修正缩放，实现模型容量扩展且初始化不变。
- 在下游任务上达到全量微调性能，原能力零退化。
- 只扩展部分层即可接近全量效果，计算成本大幅降低。
将预训练模型适配到特定任务时，常出现灾难性遗忘，即新知识覆盖原有能力。现有方法要么牺牲新任务表现，要么难以平衡训练稳定性与预训练知识的高效复用。本文提出一种保持函数形式的扩展方法：在Transformer子模块内复制预训练参数，并施加缩放校正，确保扩展后模型在初始化时数学上等同于原模型，从而实现稳定训练并充分利用已有知识。实验表明，该方法消除了可塑性与稳定性之间的权衡，在下游任务上表现与全量微调相当，且原模型能力无任何退化。此外，我们验证了方法的模块化特性——仅选择性扩展少量层，即可达到全量微调性能，计算成本仅为后者的几分之一。
原文摘要 · Abstract (English)
Adapting pre-trained models to specialized tasks often leads to catastrophic forgetting, where new knowledge overwrites foundational capabilities. Existing methods either compromise performance on the new task or struggle to balance training stability with efficient reuse of pre-trained knowledge. We introduce a novel function-preserving expansion method that resolves this dilemma. Our technique expands model capacity by replicating pre-trained parameters within transformer submodules and applying a scaling correction that guarantees the expanded model is mathematically identical to the original at initialization, enabling stable training while exploiting existing knowledge. Empirically, our method eliminates the trade-off between plasticity and stability, matching the performance of full fine-tuning on downstream tasks without any degradation of the model's original capabilities. Furthermore, we demonstrate the modularity of our approach, showing that by selectively expanding a small subset of layers we can achieve the same performance as full fine-tuning at a fraction of the computational cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。