提出GAIN方法，解决大模型跨域适应中的遗忘问题。
GAIN: Multiplicative Modulation for Domain Adaptation
- 用乘法更新权重（W_new = S * W），保持预训练矩阵列空间不变
- 在8个领域连续适应中，旧领域困惑度提升7%-13%，优于LoRA的降18%-36%
- 无需存储历史数据或正则化，适合持续学习场景
将大模型适配到新领域时会产生遗忘，因为标准方法（如全量微调、LoRA）会在权重空间注入新方向。我们证明遗忘由一个代数性质决定：更新是否保持预训练权重矩阵的列空间（命题1）。为此提出GAIN，一种最简单的乘法替代方案（W_new = S * W），其构造即满足该性质，且可无缝融入现有权重，推理成本为零。在五个模型（774M至70B）上连续适应八个领域，GAIN使早期领域困惑度提升7%-13%，而LoRA反而下降18%-36%。GAIN在不存储历史数据的情况下达到回放增强型LoRA效果，并优于EWC在遗忘-适应权衡前沿的表现。与仅能通过牺牲域内适应来减少遗忘的LoRA不同，GAIN同时实现两者，无领域边界且无需正则化。该原则具普适性：独立提出的乘法方法(IA)^3也提升了早期领域性能。
原文摘要 · Abstract (English)
Adapting LLMs to new domains causes forgetting because standard methods (e.g., full fine-tuning, LoRA) inject new directions into the weight space. We show that forgetting is governed by one algebraic property: whether the update preserves the column span of the pretrained weight matrix (Proposition 1). We propose GAIN, the simplest multiplicative alternative (W_new = S * W), which satisfies this by construction and can be absorbed into existing weights for zero inference cost. Across five models (774M to 70B) adapted sequentially over eight domains, GAIN improves earlier-domain perplexity by 7-13%, while LoRA degrades it by 18-36%. GAIN matches replay-augmented LoRA without storing prior data and dominates EWC on the forgetting-adaptation Pareto front. While LoRA can only reduce forgetting by sacrificing in-domain adaptation, GAIN achieves both with no domain boundaries and no regularization. The principle generalises: (IA)^3, an independent multiplicative method, also improves earlier domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。