通过智能扩展与解耦调优，让大模型持续学习新领域而不遗忘旧知识。
ADEPT: Continual Pretraining via Adaptive Expansion and Dynamic Decoupled Tuning
- 按通用能力重要性选择性扩展层，减少对通用知识的干扰。
- 仅用15%参数调优，通用和目标领域性能提升超5%。
- 适合需要高效、不遗忘地适应新领域的模型开发者。
传统大语言模型持续预训练在领域适应中常面临灾难性遗忘和领域容量有限的问题。现有方法采用统一层扩展，引入额外可训练参数以容纳新知识，但均匀扩展与更新仍混淆通用与领域学习，降低效果。初步研究发现，大模型存在功能分化，不同层和单元对通用能力贡献不一，提示参数扩展与优化应具备功能感知。为此，我们提出ADEPT——自适应扩展与动态解耦调优框架，分两阶段实现领域自适应的持续预训练。首先，基于通用能力引导的有选择性层扩展，复制对通用领域最不关键的层以增强表征能力，同时最小化对通用知识的干扰；其次，实施自适应单元级解耦调优，根据单元在通用领域的权重重要性进行解耦，并分配非对称学习率，平衡知识注入与保留。在数学与医疗基准测试中，ADEPT相比全参数持续预训练，在通用领域最高提升5.76%，目标领域提升5.58%，且仅需调优15%参数，训练时间不足50%。消融实验、理论分析及拓展研究进一步验证了针对性扩展与解耦优化的必要性，为高效、鲁棒的领域自适应持续预训练提供了新范式。代码已开源：https://github.com/PuppyKnightUniversity/ADEPT
原文摘要 · Abstract (English)
Conventional continual pretraining (CPT) for large language model (LLM) domain adaptation often suffers from catastrophic forgetting and limited domain capacity. Existing strategies adopt layer expansion, introducing additional trainable parameters to accommodate new knowledge. However, the uniform expansion and updates still entangle general and domain learning, undermining its effectiveness. Our pilot studies reveal that LLMs exhibit functional specialization, where layers and units differentially encode general-critical capabilities, suggesting that parameter expansion and optimization should be function-aware. We then propose ADEPT, Adaptive Expansion and Dynamic Decoupled Tuning for continual pretraining, a two-stage framework for domain-adaptive CPT. ADEPT first performs General-Competence Guided Selective Layer Expansion, duplicating layers least critical for the general domain to increase representational capacity while minimizing interference with general knowledge. It then applies Adaptive Unit-Wise Decoupled Tuning, disentangling parameter units within expanded layers according to their general-domain importance and assigning asymmetric learning rates to balance knowledge injection and retention. Experiments on mathematical and medical benchmarks show that ADEPT outperforms full-parameter CPT by up to 5.76% on the general domain and 5.58% on the target domain with only 15% of parameters tuned and less than 50% training time. Ablation studies, theoretical analysis, and extended investigations further demonstrate the necessity of targeted expansion and decoupled optimization, providing new principles for efficient and robust domain-adaptive CPT. Our code is open-sourced at https://github.com/PuppyKnightUniversity/ADEPT
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。