研究多语言模型中知识遗忘的原因，发现非拉丁语系更易遗忘。
What Causes Knowledge Loss in Multilingual Language Models?
- 用不同秩的LoRA适配器测试参数共享对多语言学习的影响。
- 52种语言实验表明，非拉丁字母语言遗忘更严重，拉丁字母语言迁移更有效。
- 揭示语言书写系统差异是影响知识保留的关键因素，适合多语言模型研究者参考。
自然语言处理中的跨语言迁移通过利用共享的语言知识提升多语言性能。然而，传统同时处理所有数据的方法难以模拟真实场景，常引发灾难性遗忘问题——在新任务上微调会损害先前任务的表现。本研究聚焦多语言环境下的这一问题，关注语言差异如何影响表征学习，而非仅限于模型参数。我们使用52种语言，通过不同秩的LoRA适配器评估非共享、部分共享和完全共享参数的效果，旨在检验适配器形式的参数共享能否缓解遗忘并保留已有知识。结果表明，使用非拉丁文字的语言更容易发生灾难性遗忘，而拉丁文字语言则促进更有效的跨语言迁移。
原文摘要 · Abstract (English)
Cross-lingual transfer in natural language processing (NLP) models enhances multilingual performance by leveraging shared linguistic knowledge. However, traditional methods that process all data simultaneously often fail to mimic real-world scenarios, leading to challenges like catastrophic forgetting, where fine-tuning on new tasks degrades performance on previously learned ones. Our study explores this issue in multilingual contexts, focusing on linguistic differences affecting representational learning rather than just model parameters. We experiment with 52 languages using LoRA adapters of varying ranks to evaluate non-shared, partially shared, and fully shared parameters. Our aim is to see if parameter sharing through adapters can mitigate forgetting while preserving prior knowledge. We find that languages using non-Latin scripts are more susceptible to catastrophic forgetting, whereas those written in Latin script facilitate more effective cross-lingual transfer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。