加点英文数据能唤醒大模型的外语能力，否则会遗忘
Emergent Abilities of Large Language Models under Continued Pretraining for Language Adaptation
- 用英语数据混合训练，让模型在目标语言中突然具备新能力
- 不用英语时，模型早期就出现灾难性遗忘，影响后续表现
- 提出课程学习和权重平滑法，可减少对英语数据的依赖
持续预训练（CPT）是将现有大语言模型适配新语言的常用方法。当前做法常混合部分英语数据，但其作用尚不明确。本文发现：加入英语不影响验证困惑度，却对目标语言下游能力的涌现至关重要。我们构建了一个与语言无关的上下文学习基准，揭示若不包含英语，模型在早期训练即出现灾难性遗忘，虽准确率未立即下降，但已导致模型参数大幅偏移，进而损害对目标语言提示的泛化能力。基于此，我们提出课程学习与权重指数移动平均（EMA）作为替代方案，有效降低对英语数据的依赖。本研究揭示了语言适配中涌现能力的动态机制，为未来更高效方法设计提供基础。
原文摘要 · Abstract (English)
Continued pretraining (CPT) is a popular approach to adapt existing large language models (LLMs) to new languages. When doing so, it is common practice to include a portion of English data in the mixture, but its role has not been carefully studied to date. In this work, we show that including English does not impact validation perplexity, yet it is critical for the emergence of downstream capabilities in the target language. We introduce a language-agnostic benchmark for in-context learning (ICL), which reveals catastrophic forgetting early on CPT when English is not included. This in turn damages the ability of the model to generalize to downstream prompts in the target language as measured by perplexity, even if it does not manifest in terms of accuracy until later in training, and can be tied to a big shift in the model parameters. Based on these insights, we introduce curriculum learning and exponential moving average (EMA) of weights as effective alternatives to mitigate the need for English. All in all, our work sheds light into the dynamics by which emergent abilities arise when doing CPT for language adaptation, and can serve as a foundation to design more effective methods in the future.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。