LoKI让大模型学新任务时少忘旧知识，兼顾效果与记忆。
LoKI: Low-damage Knowledge Implanting of Large Language Models
- 基于变压器架构的知识存储机制，设计低损伤微调方法
- 在多种模型上保持更强通用能力，任务性能不逊全量微调
- 适合需长期学习新任务又怕遗忘的老知识的场景
微调能适应特定任务，但存在灾难性遗忘（CF）风险，即预训练阶段的关键知识被覆盖。为解决通用框架下的CF问题，我们提出低损伤知识植入（LoKI），一种参数高效微调（PEFT）技术，利用对大语言模型知识存储机制的最新理解。我们在两个真实微调场景中对比了LoKI与最先进的PEFT方法。结果表明，LoKI在保留通用能力方面表现显著更优，同时其任务特定性能与全参数微调及现有PEFT方法相当甚至更优，涵盖多种模型架构。本工作将大语言模型知识存储的机制洞察与实际微调目标结合，实现了任务适配与通用能力保留之间的有效平衡。
原文摘要 · Abstract (English)
Fine-tuning adapts pretrained models for specific tasks but poses the risk of catastrophic forgetting (CF), where critical knowledge from pretraining is overwritten. To address the issue of CF in a general-purpose framework, we propose Low-damage Knowledge Implanting (LoKI), a parameter-efficient fine-tuning (PEFT) technique that utilizes recent mechanistic understanding of how knowledge is stored in transformer architectures. We compare LoKI against state-of-the-art PEFT methods in two real-world fine-tuning scenarios. The results show that LoKI demonstrates significantly better preservation of general capabilities. At the same time, its task-specific performance is comparable to or even surpasses that of full parameter fine-tuning and these PEFT methods across various model architectures. Our work bridges the mechanistic insights of LLMs' knowledge storage with practical fine-tuning objectives, enabling an effective balance between task-specific adaptation and the retention of general-purpose capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。