用少量源语言数据保护关键参数，防止大模型适配新语言时遗忘旧知识。
Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updates
- 通过重要性评分筛选关键参数，用列冻结策略保护源语言能力。
- 7B和13B模型在目标语言上遗忘率分别降至3.4%和2.8%，远低于全微调的20.3%和22.3%。
- 适合资源有限下需多语言扩展的大模型应用，如低资源语种适配。
扩展指令型大语言模型的语言多样性对全球可用性至关重要，但常受制于昂贵的目标语言标注数据及适应过程中的灾难性遗忘。本文在低资源现实约束下，仅使用未标注的目标语言数据来适配指令型大模型。提出源语言保护更新（SSU）策略，通过小量源数据和参数重要性评分，识别维持源语言能力的关键参数，并采用列级冻结策略在适配前保护这些参数。在五种语系差异大的语言及7B和13B模型上的实验表明，SSU有效缓解灾难性遗忘：7B与13B模型在单语源任务上的性能下降平均仅为3.4%和2.8%，显著优于全微调的20.3%和22.3%。同时，目标语言表现接近甚至超越全微调，在7B模型上所有基准均更优，13B模型则在多数基准上领先。
原文摘要 · Abstract (English)
Expanding the linguistic diversity of instruct large language models (LLMs) is crucial for global accessibility but is often hindered by the reliance on costly specialized target language labeled data and catastrophic forgetting during adaptation. We tackle this challenge under a realistic, low-resource constraint: adapting instruct LLMs using only unlabeled target language data. We introduce Source-Shielded Updates (SSU), a selective parameter update strategy that proactively preserves source knowledge. Using a small set of source data and a parameter importance scoring method, SSU identifies parameters critical to maintaining source abilities. It then applies a column-wise freezing strategy to protect these parameters before adaptation. Experiments across five typologically diverse languages and 7B and 13B models demonstrate that SSU successfully mitigates catastrophic forgetting. It reduces performance degradation on monolingual source tasks to just 3.4% (7B) and 2.8% (13B) on average, a stark contrast to the 20.3% and 22.3% from full fine-tuning. SSU also achieves target-language performance highly competitive with full fine-tuning, outperforming it on all benchmarks for 7B models and the majority for 13B models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。